{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_3a0i46j",
    "id": "44C527C86C2941789DE841DE4A782754",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# 过拟合、欠拟合及其解决方案\n",
    "1. 过拟合、欠拟合的概念\n",
    "2. 权重衰减\n",
    "3. 丢弃法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_2rrhvg5",
    "id": "1C7884059BD2464DB639268E872E99C2",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# 模型选择、过拟合和欠拟合\n",
    "\n",
    "## 训练误差和泛化误差\n",
    "在解释上述现象之前，我们需要区分训练误差（training error）和泛化误差（generalization error）。通俗来讲，前者指模型在训练数据集上表现出的误差，后者指模型在任意一个测试数据样本上表现出的误差的期望，并常常通过测试数据集上的误差来近似。计算训练误差和泛化误差可以使用之前介绍过的损失函数，例如线性回归用到的平方损失函数和softmax回归用到的交叉熵损失函数。\n",
    "\n",
    "机器学习模型应关注降低泛化误差。\n",
    "\n",
    "## 模型选择\n",
    "### 验证数据集\n",
    "从严格意义上讲，测试集只能在所有超参数和模型参数选定后使用一次。不可以使用测试数据选择模型，如调参。由于无法从训练误差估计泛化误差，因此也不应只依赖训练数据选择模型。鉴于此，我们可以预留一部分在训练数据集和测试数据集以外的数据来进行模型选择。这部分数据被称为验证数据集，简称验证集（validation set）。例如，我们可以从给定的训练集中随机选取一小部分作为验证集，而将剩余部分作为真正的训练集。\n",
    "\n",
    "### K折交叉验证  \n",
    "由于验证数据集不参与模型训练，当训练数据不够用时，预留大量的验证数据显得太奢侈。一种改善的方法是K折交叉验证（K-fold cross-validation）。在K折交叉验证中，我们把原始训练数据集分割成K个不重合的子数据集，然后我们做K次模型训练和验证。每一次，我们使用一个子数据集验证模型，并使用其他K-1个子数据集来训练模型。在这K次训练和验证中，每次用来验证模型的子数据集都不同。最后，我们对这K次训练误差和验证误差分别求平均。\n",
    "## 过拟合和欠拟合\n",
    "接下来，我们将探究模型训练中经常出现的两类典型问题：\n",
    "- 一类是模型无法得到较低的训练误差，我们将这一现象称作欠拟合（underfitting）；\n",
    "- 另一类是模型的训练误差远小于它在测试数据集上的误差，我们称该现象为过拟合（overfitting）。\n",
    "在实践中，我们要尽可能同时应对欠拟合和过拟合。虽然有很多因素可能导致这两种拟合问题，在这里我们重点讨论两个因素：模型复杂度和训练数据集大小。\n",
    "\n",
    "### 模型复杂度\n",
    "为了解释模型复杂度，我们以多项式函数拟合为例。给定一个由标量数据特征$x$和对应的标量标签$y$组成的训练数据集，多项式函数拟合的目标是找一个$K$阶多项式函数\n",
    "\n",
    "\n",
    "$$\n",
    " \\hat{y} = b + \\sum_{k=1}^K x^k w_k \n",
    "$$\n",
    "\n",
    "\n",
    "来近似 $y$。在上式中，$w_k$是模型的权重参数，$b$是偏差参数。与线性回归相同，多项式函数拟合也使用平方损失函数。特别地，一阶多项式函数拟合又叫线性函数拟合。\n",
    "\n",
    "给定训练数据集，模型复杂度和误差之间的关系：\n",
    "\n",
    "![Image Name](https://cdn.kesci.com/upload/image/q5jc27wxoj.png?imageView2/0/w/960/h/960)\n",
    "\n",
    "### 训练数据集大小\n",
    "影响欠拟合和过拟合的另一个重要因素是训练数据集的大小。一般来说，如果训练数据集中样本数过少，特别是比模型参数数量（按元素计）更少时，过拟合更容易发生。此外，泛化误差不会随训练数据集里样本数量增加而增大。因此，在计算资源允许的范围之内，我们通常希望训练数据集大一些，特别是在模型复杂度较高时，例如层数较多的深度学习模型。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_rxqm2hz",
    "id": "662A794758DD42EFA08B79C1EF032C65",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# 多项式函数拟合实验"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "graffitiCellId": "id_yjb9esq",
    "id": "F39C60E42E09441D8602243557564E0A",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.2.0\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import torch\n",
    "import numpy as np\n",
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "import d2lzh1981 as d2l\n",
    "print(torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_1aakii8",
    "id": "193FB89ACE2E4D4888FB258A250939AD",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 初始化模型参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
    "graffitiCellId": "id_3xv7jo8",
    "id": "9714C0B5EF3344AB89F6C05008E10951",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "n_train, n_test, true_w, true_b = 100, 100, [1.2, -3.4, 5.6], 5\n",
    "features = torch.randn((n_train + n_test, 1))\n",
    "poly_features = torch.cat((features, torch.pow(features, 2), torch.pow(features, 3)), 1) \n",
    "labels = (true_w[0] * poly_features[:, 0] + true_w[1] * poly_features[:, 1]\n",
    "          + true_w[2] * poly_features[:, 2] + true_b)\n",
    "labels += torch.tensor(np.random.normal(0, 0.01, size=labels.size()), dtype=torch.float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "graffitiCellId": "id_m2mu8wo",
    "id": "FDF06E080C72455296FF40E5A03C8A1D",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[ 0.1538],\n",
       "         [-0.3109]]),\n",
       " tensor([[ 0.1538,  0.0237,  0.0036],\n",
       "         [-0.3109,  0.0966, -0.0300]]),\n",
       " tensor([5.1352, 4.1344]))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features[:2], poly_features[:2], labels[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_6fb98pt",
    "id": "04AB33FA366C42CDBF58FE28874F3F27",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 定义、训练和测试模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "graffitiCellId": "id_tq07955",
    "id": "DD5CB5E380BE4BF498E4F3D5A9F97F43",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def semilogy(x_vals, y_vals, x_label, y_label, x2_vals=None, y2_vals=None,\n",
    "             legend=None, figsize=(3.5, 2.5)):\n",
    "    # d2l.set_figsize(figsize)\n",
    "    d2l.plt.xlabel(x_label)\n",
    "    d2l.plt.ylabel(y_label)\n",
    "    d2l.plt.semilogy(x_vals, y_vals)\n",
    "    if x2_vals and y2_vals:\n",
    "        d2l.plt.semilogy(x2_vals, y2_vals, linestyle=':')\n",
    "        d2l.plt.legend(legend)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "graffitiCellId": "id_3lzy7bj",
    "id": "3DA4335947C44C018CD2C8B215A641ED",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "num_epochs, loss = 100, torch.nn.MSELoss()\n",
    "\n",
    "def fit_and_plot(train_features, test_features, train_labels, test_labels):\n",
    "    # 初始化网络模型\n",
    "    net = torch.nn.Linear(train_features.shape[-1], 1)\n",
    "    # 通过Linear文档可知，pytorch已经将参数初始化了，所以我们这里就不手动初始化了\n",
    "    \n",
    "    # 设置批量大小\n",
    "    batch_size = min(10, train_labels.shape[0])    \n",
    "    dataset = torch.utils.data.TensorDataset(train_features, train_labels)      # 设置数据集\n",
    "    train_iter = torch.utils.data.DataLoader(dataset, batch_size, shuffle=True) # 设置获取数据方式\n",
    "    \n",
    "    optimizer = torch.optim.SGD(net.parameters(), lr=0.01)                      # 设置优化函数，使用的是随机梯度下降优化\n",
    "    train_ls, test_ls = [], []\n",
    "    for _ in range(num_epochs):\n",
    "        for X, y in train_iter:                                                 # 取一个批量的数据\n",
    "            l = loss(net(X), y.view(-1, 1))                                     # 输入到网络中计算输出，并和标签比较求得损失函数\n",
    "            optimizer.zero_grad()                                               # 梯度清零，防止梯度累加干扰优化\n",
    "            l.backward()                                                        # 求梯度\n",
    "            optimizer.step()                                                    # 迭代优化函数，进行参数优化\n",
    "        train_labels = train_labels.view(-1, 1)\n",
    "        test_labels = test_labels.view(-1, 1)\n",
    "        train_ls.append(loss(net(train_features), train_labels).item())         # 将训练损失保存到train_ls中\n",
    "        test_ls.append(loss(net(test_features), test_labels).item())            # 将测试损失保存到test_ls中\n",
    "    print('final epoch: train loss', train_ls[-1], 'test loss', test_ls[-1])    \n",
    "    semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',\n",
    "             range(1, num_epochs + 1), test_ls, ['train', 'test'])\n",
    "    print('weight:', net.weight.data,\n",
    "          '\\nbias:', net.bias.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_tvv0n31",
    "id": "FEBABF4CE4FC49F8869E557513F6CFD8",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 三阶多项式函数拟合（正常）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "graffitiCellId": "id_pc28vr5",
    "id": "CD685B472B744329A1CFC47C9F0B5E89",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "final epoch: train loss 0.00010211031622020528 test loss 0.00010742605081759393\n",
      "weight: tensor([[ 1.1953, -3.3977,  5.6016]]) \n",
      "bias: tensor([4.9978])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEGCAYAAAB7DNKzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3hUZdr48e89k0Y6hFASiKFJ7y2IXVFQWVZEOiIiwba678++lq2u7L67vupaaMYGUhRXUVEQRREFgdAMBAid0GsS0svz++NMNGASEpjJmUzuz3XNRU6ZZ+7jibnnKed5xBiDUkopVRGH3QEopZTybpoolFJKVUoThVJKqUppolBKKVUpTRRKKaUq5Wd3AJ7QsGFDEx8fb3cYSilVqyQnJx83xkSfu98nE0V8fDxr1661OwyllKpVRGRvefu16UkppVSlNFEopZSqlCYKpZRSlfLJPgqllKquwsJC0tPTycvLszsUjwsKCqJZs2b4+/tX6XxNFEopBaSnpxMWFkZ8fDwiYnc4HmOM4cSJE6Snp9OiRYsqvcenmp5EZLCITM/IyLA7FKVULZOXl0dUVJRPJwkAESEqKqpaNSefShTGmE+MMYkRERF2h6KUqoV8PUmUqu51+lSiuFhfpR5hxvJddoehlFJeRRNFGd9sO8YLX24nr7DY7lCUUnXQ6dOnee2116r9vptuuonTp097ICKLJooyru/QmNzCYn7YedzuUJRSdVBFiaK4uPIvr4sWLSIyMtJTYWmiKKtf5CleD/wP323aYXcoSqk66IknnmDnzp1069aN3r17c8011zB69Gg6d+4MwG9/+1t69uxJx44dmT59+s/vi4+P5/jx4+zZs4f27dszadIkOnbsyA033EBubu5Fx6XDY8sIKCngCudmPtq2lpKSy3A46kbHllLqbH/+ZDNbDma6tcwOMeH8cXDHSs+ZMmUKKSkpbNiwgW+++Yabb76ZlJSUn4exJiUl0aBBA3Jzc+nduze33XYbUVFRZ5WRlpbGnDlzmDFjBsOHD2fBggWMHTv2omLXGkVZTTrx1U3LWJzdmk0HdIitUspeffr0OetZh5dffpmuXbuSkJDA/v37SUtL+9V7WrRoQbdu3QDo2bMne/bsueg4vL5GISItgaeACGPMME9/3lUd4nA6trNp9bd0az7E0x+nlPJC5/vmX1NCQkJ+/vmbb75h6dKlrFy5kuDgYK6++upyn4UIDAz8+Wen0+mWpidbahQikiQiR0Uk5Zz9A0Vkm4jsEJEnAIwxu4wxE2sqtsjgAH4fvY47froD9q2qqY9VSinCwsLIysoq91hGRgb169cnODiYrVu3smpVzf19sqvp6S1gYNkdIuIEXgUGAR2AUSLSoeZDg9Dut/J04QT212tnx8crpeqoqKgo+vfvT6dOnXj00UfPOjZw4ECKioro0qULzzzzDAkJCTUWly1NT8aY5SISf87uPsAOY8wuABGZCwwBtlSlTBFJBBIB4uLiLiq+qzvF8+cvBtBq+ykmcARO7oK2gy6qTKWUqor33nuv3P2BgYF8/vnn5R4r7Ydo2LAhKSm/NNQ88sgjbonJmzqzY4H9ZbbTgVgRiRKRqUB3EXmyojcbY6YbY3oZY3pFR/9qJb9qadEwhNaNQvlyyxH47gX4+AHIL786qJRSvs6bEkV5Y1GNMeaEMeYeY0wrY8zzlRbgxkkBb+nSlB92nmBq5EMwfiEEhl10mUopVRt5U6JIB5qX2W4GHKxOAe6cFPD+a1ozpFsMU5bs5t8b/TDGwOoZ8NVfLrpspZSqTbxpeOwaoI2ItAAOACOB0dUpQEQGA4Nbt2590cH4Ox28MLwb9fyd/OfrHeTkF/G0SUGyj0FxETi96T+dUkp5jl3DY+cAK4G2IpIuIhONMUXAA8BiIBWYb4zZXJ1y3T3NuNMhPD+0MxP6x/PG93v4KxMxw9+2kkSJThyolKob7Br1NKqC/YuARRdarjtrFGXK5NlbOiAISd/vBvHjmRvikPdGQIffQN/JbvsspZTyRt7UR3HRPLVwkYjwzC3tufOyeJK+383fF+/EhERBcNT536yUUlV0odOMA7z44ovk5OS4OSKLTyUKTxIR/ji4A3deFs+MH9L5W73HMZ1usw5mVqvPXSmlyuWticKnemQ90fR0Tvn8cbD1sPgb3++hBHi2rxOZeR3c8FfodZdHPlcpVTeUnWZ8wIABNGrUiPnz55Ofn8+tt97Kn//8Z7Kzsxk+fDjp6ekUFxfzzDPPcOTIEQ4ePMg111xDw4YNWbZsmVvj8qkaRU2smV2aLCb0j+fN7/fw15X5mN53w6X65LZSPuXNm2H9bOvn4kJre+M8a7sgx9pOWWBt52VY21sWWtvZJ6ztba4nqbOOVOkjp0yZQqtWrdiwYQMDBgwgLS2N1atXs2HDBpKTk1m+fDlffPEFMTExbNy4kZSUFAYOHMiDDz5ITEwMy5Ytc3uSAB9LFDWltIP77stbkLQynWeyh1ES2gSMgQ1zrF8qpZS6CEuWLGHJkiV0796dHj16sHXrVtLS0ujcuTNLly7l8ccf57vvvsOTX4xLadPThX8WT93cHj+ng6nf7qSwyPD3Hhk4P7oHMNCtWo+AKKW8zYTPfvnZ6X/2dkDw2dtBEWdvh0SdvR3WuNofb4zhySefZPLkX4+sTE5OZtGiRTz55JPccMMNPPvss9Uuvzp8qkZRE01PZYkIjw9sy4PXtWHe2v08uiaMojs+ha7ljv5VSqlKlZ1m/MYbbyQpKYkzZ84AcODAAY4ePcrBgwcJDg5m7NixPPLII6xbt+5X73U3n6pR2EFE+H8DLiXAKfxryXZyC5vwUpwhIOcQfPYw3PLiBX2bUErVPWWnGR80aBCjR4+mX79+AISGhjJr1ix27NjBo48+isPhwN/fn9dffx2AxMREBg0aRNOmTd3eTyHGGLcWaKcyTU+Tylsi0NNmfreLv32WylWXRjP96mICPxwPI2ZD8941HotSqnpSU1Np37693WHUmPKuV0SSjTG9zj1Xm57c6O4rWjJlaGeWpx1j3BLImrz2lyShHdxKqVrKpxKFNxjZJ46XR3Zn3b5T3PHuT2TmFcLmj2Dq5fpgnlKqVtJE4QGDu8bwyuge/JSewbg3VnPGLxIiL4F69e0OTSlVCV9qiq9Mda9TE4WHDOzUhNfG9GDLwQzGfOlHxq2zwb+eNUX5se12h6eUOkdQUBAnTpzw+WRhjOHEiRMEBQVV+T3ame1hS7cc4b7Z62jVKJR3J/ah4dePwNZF8OA6a+y1UsorFBYWkp6eTl5ent2heFxQUBDNmjXD39//rP0VdWb7VKIo1atXL7N27Vq7w/jZ8u3HSHx3LTGR9Zh7a30aZadB52F2h6WUUmepE6OevNWVl0bz7sS+HMvM59b5J9jT1DUvVF6mvYEppVQVaKKoIb3jGzAnMYHcwmJun7aSPZu+g5e6wM6v7Q5NKaUqpYmiBnWKjWD+5AScIoz48DSnG/eD6HZ2h6WUUpXSRFHDWjcK4/17+hEYHEr/XeNZeSzQmnX2+A67Q1NKqXJporBB8wbBvH9PP2Ii63Hnm6vZ8fEU64G8494xUksppcryqUQhIoNFZHpGRobdoZxX4/Ag5k3uR5vGoYxZE09qu/shyvPToyulVHX5VKKwe66n6moQEsB7kxJo3uwSbk7uwYfrD0DmIVhwN2Qftzs8pZQCfCxR1EbhQf68M7EP/VpF8cj7G0le+TXsXKbzQimlvIYmCi8QHODHjDt60T2uPiOX12f5zcugaRfrYPYJe4NTStV5mii8RHCAH0l39ubSxmFMmrOZH3edsBZqf6krHFhnd3hKqTpME4UXiajnzzt39aF5g2DufHMNa4paQ5fh0KiD3aEppeowTRReJio0kDmTEohrEMyY+XtZ1voJ8A+CwlzY+tn5C1BKKTfTROGFosMCmZOYwKWNQ0l8dy1fpByGla/C3DFwbJvd4Sml6hivTxQiEiIib4vIDBEZY3c8NaVBSACz706gU2wE97+3jk/ChsHYBRDd1u7QlFJ1jC2JQkSSROSoiKScs3+giGwTkR0i8oRr91DgA2PMJOA3NR6sjSLq+fPuxL70vKQ+D83fzAcZriRxaBPsW2VvcEqpOsOuGsVbwMCyO0TECbwKDAI6AKNEpAPQDNjvOq24BmP0CqGBfrw9oQ+XtWrII+9v5L1Ve+GTB2HRI1BSYnd4Sqk6wJZEYYxZDpw8Z3cfYIcxZpcxpgCYCwwB0rGSBVQSr4gkishaEVl77NgxT4Rtm3oBTmaO78U1baP5w0cpvN/ybzD6fXB4fcuhUsoHeNNfmlh+qTmAlSBigQ+B20TkdeCTit5sjJlujOlljOkVHR3t2UhtEOTvZOq4ntzYsTGPLs3g9XU51oFvpkB6sr3BKaV8mp/dAZQh5ewzxphsYEKVCvhlzWy3BuYtAv2cvDK6Bw/P38g/vtiKyT3NvdvnIAVnoFlPu8NTSvkob6pRpAPNy2w3A6o14VFtmxTwQvg7HfzfiG7c3rMZ//z2MK+0mo657k/WwbwMa20LpZRyI29KFGuANiLSQkQCgJHAwuoUUJumGb8YTofwj9u6MLpvHP9ecZwpi9MweZkwcwB89We7w1NK+Ri7hsfOAVYCbUUkXUQmGmOKgAeAxUAqMN8Ys7k65daFGkUph0N47reduKPfJUxbvou/LtmHaXsTtLrO7tCUUj7Glj4KY8yoCvYvAhZdaLm+3kdxLhHhz7/piJ/DQdL3uylIuI2/XNLJyv5bP4PYnhDWxO4wlVK1nDc1PV20ulSjKCUiPHNLe+69uhWzVu3jsQWbKM45DR/dB0u1GUopdfG8adSTukAiwmM3tiXIz8n/Ld1OflEJL4xbiH9UvN2hKaV8gE/VKOpKZ3Z5RISHrm/DE4Pa8cnGg9z/VSH5fqFQlA+fPQyn9todolKqlvKpRFEXm57Odc9VrfjT4A4s2XKExHeSyTu+F1IWwO5v7Q5NKVVLadOTD7qzfwvqBTh54sOfuHNhMW8kriakfiO7w1JK1VI+VaOoy01P5xrRO44XR3Rj9e6TTJi/k+z8IjiyBWYPh9zTdoenlKpFfCpRaNPT2YZ0i+Wlkd1Zu+ckE95aQ97xPXAsFXLPnY9RKaUqpk1PPm5w1xgAfj9vA3eY+iRNWkVoSIh1sCgf/AJtjE4pVRv4VI1Cm57KN7hrDC+N7EbyvlOMf2cjmXmFsOJFSLoR8jLtDk8p5eV8KlFo01PFbukSwyujurNx/2nGzfyR7Ig20KgD+AfbHZpSysv5VKJQlRvUuSlTx/Yk9VAWt38dzskBL4LTz6pVFOTYHZ5Syktpoqhjru/QmOl39GTnsTOMnL6SoxnZMGsozL5dpyhXSpVLE0UddHXbRrx5Z2/ST+UyYsYaTnR/APpOBilv7SilVF3nU4lCO7Or7rLWDXnnrj4cz8pnyNJw9je53jqw7QvYsdTe4JRSXsWnEoV2ZldPr/gGzJ7UlzP5RQyftpLdx87A9y9a63CXlNgdnlLKS/hUolDV16VZJO/dnUB+UQnDp69i53XTYdQ8cOivhlLKon8NFB1iwpmXmADA8He2kZrpb9UoFj5oTSiolKrTNFEoANo0DmNeYgL+TgejZqxi8/6jcGIHHE+zOzSllM00UaiftYwOZd7kBEIC/Bj15kY2XvMWXP2EdbCowNbYlFL20UShznJJVAhzExOICPZnzJvrSd57Ek7vh9f6QuqndoenlLKBTyUKHR7rHs0bBDMvsR8NQwMY98ZqVh8qhqg2EBFrd2hKKRv4VKLQ4bHuExNZj/mT+xEbWY9xs1P5tverENPdOph93N7glFI1yqcShXKvRuFBzE1MoGV0KJPeXsuXW45Yo6Be6gZHNtsdnlKqhmiiUJWKCg1k7qQE2seEc++sZL7MbgWdhkJUa7tDU0rVEE0U6rwigv2ZNbEP3eMimfzRARbEPmYteFSQA7u+sTs8pZSHaaJQVRIW5M/bd/WhX6soHn5/I7N/3AvL/wmzhsHpfXaHp5TyIE0UqsqCA/x4Y3xvrm3XiKf+m8KsgOEwcjZExtkdmlLKgzRRqGoJ8ncydWxPbuzYmKcX7Wb6YVdfxcH1sOMre4NTSnmEn90BnI+ItASeAiKMMcPsjkdBgJ+DV0b34H/mbeDvi7aSV1DM7/b/Ack5CS1/AIfT7hCVUm7k0RqFiCSJyFERSTln/0AR2SYiO0TkicrKMMbsMsZM9GScqvr8nQ5eHNGNoT1ieWFpGv8K/wMlYxZoklDKB3m6RvEW8ArwTukOEXECrwIDgHRgjYgsBJzA8+e8/y5jzFEPx6gukJ/Twb+GdaVBcACvrtjN/oJj/Ov2GAKW/x2adoX2g+0OUSnlBh5NFMaY5SISf87uPsAOY8wuABGZCwwxxjwP3HKhnyUiiUAiQFycdq7WFIdDeOrm9jQMC2TK51vJyclmeskyHHkZmiiU8hF2dGbHAvvLbKe79pVLRKJEZCrQXUSerOg8Y8x0Y0wvY0yv6Oho90WrzktEuOeqVkwZ2pmlaRlMlj+Sd91z1sH8LHuDU0pdNDsShZSzz1R0sjHmhDHmHmNMK1eto+KCdVJAW43sE8eUoZ35ckcW9763gfzsUzBzAHz9nN2hKaUugh2JIh1oXma7GXDQHQXrpID2G9knjr/f2pll245x3/ztFLW4GuIvtzsspdRFsCNRrAHaiEgLEQkARgIL3VGw1ii8w+i+cTx3aye+2nacuw4PJa+5K1EcXG8tsaqUqlU8PTx2DrASaCsi6SIy0RhTBDwALAZSgfnGGLdMRao1Cu8xpu8l/PO2LnyXdoy7315L3sEtMPN6+OElu0NTSlWTGFNh90CtIyKDgcGtW7eelJamaz17gw/XpfPI+xvpHd+At7ulEtT1NgjSRK6UNxKRZGNMr3P3+9QUHlqj8D5DezTjxZHdSd57ittXX8qJoiAoKYYvntTJBJWqJXwqUSjv9JuuMUy/oyfbj2Rx+9SVHNm5Ada9C3t/sDs0pVQVVClRiMhDIhIuljdEZJ2I3ODp4KpLO7O917XtGjPr7r4cO5PPbz84zf5xK6DrSOtgzkl7g1NKVaqqNYq7jDGZwA1ANDABmOKxqC6QNj15t97xDZibmEBuYTEjZu1k/8kcOLET/tMTkt+2OzylVAWqmihKH5K7CXjTGLOR8h+cU6pSHWMimDWxL9kFxYyasYp00xA63QYtrrQ7NKVUBaqaKJJFZAlWolgsImGA1w2I16an2qFTrJUsMnILGZWUTPplf4EGLayDaV+CD43EU8oXVDVRTASeAHobY3IAf6zmJ6+iTU+1R+dmrmSRU8iIaavYeyIbdi6D2cNg03y7w1NKlVHVRNEP2GaMOS0iY4GnAf3ari5K1+aRvDcpgZyCIoZPW8nOsF4w7E3orOtTKeVNqpooXgdyRKQr8BiwlzJrTHgLbXqqfTrFRjA3sR/FJYYR01exteH11uJHOSfhk4d09lmlvEBVE0WRsR7hHgK8ZIx5CQjzXFgXRpueaqe2TcKYm9gPp0MYOX0Vm9JPw4F18NMHcDjl/AUopTyqqokiy7UWxDjgM9cqdf6eC0vVNa0bhfL+5MsIDfRj9IwfWePfAx7aBJf0s07QDm6lbFPVRDECyMd6nuIw1kJD/+uxqFSdFBcVzPv39KNReCDj3viRFQddySHtS0gaCLmn7A1QqTqqSonClRxmAxEicguQZ4zxuj4KVfs1jajHvMR+xEeFcNfba1i29ag1N5Qp0VqFUjap6hQew4HVwO3AcOBHEfG6oSname0bosMCmTMpgUsbh5L47loWF3WDuxZDcAMraeRl2h2iUnVKlaYZF5GNwABjzFHXdjSw1BjT1cPxXZBevXqZtWvX2h2GukgZuYWMT1rNTwcyeHFENwZ3jbFGQh3aBBM+B/8gu0NUyqdc7DTjjtIk4XKiGu9V6oJE1PNn1t196RlXn4fmrmdBcjpcOgja3axJQqkaVNU/9l+IyGIRuVNE7gQ+AxZ5LiylLKGBfrx1V28ua9WQRz7YyHunO8CVj1gHT++Hgmx7A1SqDqhqZ/ajwHSgC9AVmG6MedyTgSlVKjjAj5nje3H1pdH84b8/kbRiNxTmwVs3wUf32h2eUj7Pr6onGmMWAAs8GItSFQrydzJ1XE8emrOBv3y6hez8Ih647o9Iw0vtDk0pn1dpjUJEskQks5xXlojo0BNVowL9nLwyujtDu8fy7y+3M2V/R0yTztbBbV9AYa69ASrloyqtURhjvG6ajsqIyGBgcOvWre0ORXmIn9PBv27vSkigH9OW7yIzr5C/XRGMc+4ouOapX/ovlFJuU6XhsbWNDo/1fcYY/rVkG68u28nAjk14udcxAlpdAQEh1oSCQZHg0IF5SlXHxQ6PVcqriAiP3tiOZ2/pwBebD3PHdxFklQRAcRHMvh0+nGR3iEr5DE0Uqla76/IWvDSyG2v3nGLsG6vJyC+BTkN1TQul3EgThar1hnSLZerYnmw5mMEdb64ho1sitB1kHdyzwmqKUkpdME0Uyidc36Exr41xJYuk1WTkFkJeBswdDZ8/Znd4StVqmiiUzxhQJlmMe+NHTpfUg1FzYdA/7Q5NqVpNE4XyKQM6NGbq2J5sPZzFyOmrOB7V0zXrbAl8929d00KpC1ArEoWI/FZEZojIxyJyg93xKO92XfvGJI3vzZ4T2YyYtpLDGXlwJAW+mQJbFtodnlK1jscThYgkichREUk5Z/9AEdkmIjtE5InKyjDGfGSMmQTcibXanlKVurxNQ965qy9HMvMZPm0l6UGtYfJy6HGHdUJJib0BKlWL1ESN4i1gYNkdrjW3XwUGAR2AUSLSQUQ6i8in57walXnr0673KXVefVo0YNbdfTmdU8CIaavY44gDEchIh6mXw74f7Q5RqVrB44nCGLMcOHd8Yh9ghzFmlzGmAJgLDDHG/GSMueWc11Gx/AP43BizrrzPEZFEEVkrImuPHTvm2YtStUa35pG8NymBnIIihk9byY6jZ6CkCJx+EFirZqhRyjZ29VHEAvvLbKe79lXkd8D1wDARuae8E4wx040xvYwxvaKjo90Xqar1OsVGMG9yP0oMjJi2ki25DWDSN9C4g3VC6ifWtOVKqXLZlSiknH0VTjpljHnZGNPTGHOPMWZqhYXqmtmqApc2DmP+5AQC/RyMnL6Sdemu35HjaTBvHHw7xd4AlfJidiWKdKB5me1mwMGLLdQY84kxJjEiIuJii1I+qGV0KPPv6Uf9kADGzvyRH3Yeh4Zt4K7F1syzSqly2ZUo1gBtRKSFiAQAI4GLHreoNQp1Ps3qB/P+5H40q1+PO99cw+LNhyGuLzj9IT8Llv8vlBTbHaZSXqUmhsfOAVYCbUUkXUQmGmOKgAeAxUAqMN8Ys/liP0trFKoqGoUHMS+xHx1jwrl3VjLz1uyzDmz7ApY9D+lr7A1QKS/jU+tRlFm4aFJaWprd4Sgvl1NQxL2z1vHt9mM8NrAt913dGo5th2jX8qrGWMNplaoj6sR6FFqjUNURHODHzPG9GNIthn9+sY0XvtyOadjGOrhvFbw9GLJP2BukUl6g0qVQlfJ1/k4HLwzvRoDTwctfpVFSYnj4hkuRnBOQe1prFErhY4lC18xWF8LpEP5xWxecDuGVZTsoLCnhiYE3IZcOBIfT6txem2RN/+EXaHe4StU4bXpSCnA4hL/f2pmxCXFM+3YXz3ycQknp/x47l8GiR2D7YnuDVMomPlWjUOpiOBzCX4d0IiTQj2nf7iIzt4h/D++Kf5vrYdLXENPD7hCVsoVP1Sj0OQp1sUSEJwe157GBbVm48SCT300mp6AIYnta/RVHU+HzJ3T2WVWn+FSi0KYn5S73Xd2a527txDfbjjK8dE0LsJqhNn8ImQfsDVCpGuRTiUIpdxrT9xLeGN+b3ceyGfLqClIOZEDCvXDfKohsfv4ClPIRPpUotOlJuds17Rrxwb2X4RRh+LSVrNp90lpaFeD7l+GH/9gboFI1wKcShTY9KU9o3zScj+7vT2xkPSa8uYbVu09aT20fXAcH1lk/K+XDfCpRKOUpjcKDmD2pLzGRQdz55mrW7j0FQ2fC0OlWJ/fxNOullA/SRKFUFTUKC2LOpASahAcxPmk1K/dkWLPOAnz5LLx5ExQV2BukUh6giUKpamgUHsR7kxJoGlmP8Umr+fynQ9aBwS/BsDfAL8DeAJXyAJ9KFNqZrWpCk4ggPrinH51iw7nvvXXMWrUXQhtBiyutE36cDkuetjdIpdzIpxKFdmarmhIZHMDsuxO4tm0jnv4ohX9+sZWSElen9smdcGInFBfZG6RSbqJTeCh1geoFOJk2rifPfLyZ177Zyd4TOfx7eFeCbnjO6uB2OK0nuB0+9X1M1UH6G6zURfBzOvj7rZ34w03tWJRyiFEzVnEit9hKEoW58PYtsH623WEqdVE0USh1kUSExCtb8fqYHqQeymT4tJUcysiFkiLwrweBoXaHqNRF0UShlJsM7NSUd+7qy5HMfIa9vpI9WQ4Y8wF0GGKdkPYlnN5vb5BKXQBNFEq5UZ8WDZgzKYGcgiKGTV1J6uEs60BRPnz8AHz+uL0BKnUBfCpR6PBY5Q06N4tg/uR++DmE4VNX8sPO49bKeBOXwKAp1klF+VCQY2+gSlWRTyUKHR6rvEWbxmEsuO8ymkRYT3Ev3HgQ6l8CkXHW3FAfJsKs23QIraoVfCpRKOVNYiPr8cE9l9G9eX0enLOe6ct3Yoyxhs52GGK9nDpCXXk/TRRKeVBEsD/vTOzDzZ2b8vdFW/nTws0UlxjoNBQS7rFOOvwTnNpja5xKVUa/zijlYUH+Tv4zqjsxkUHM+G43BzPyeHlkd+oFuB7IW3A3BITA3V9ZtQ2lvIwmCqVqgMMhPHVzB2Ij6/HnT7cwasYqku7sTYOQABj+jtVvoUlCeSltelKqBt3ZvwWvj+nJlkOZDHv9B/afzIHottConXXCt/8Lq17XxZCUV9FEoVQNG9ipCbPv7svxM/kMff0HNh90DecuKYFDG+DQJnsDVOocXp8oRKS9iEwVkQ9E5F6741HKHXrHN+CDey/7+VmLZVuPWpMHDn/XWttCBMzM9gMAABNkSURBVDIPQdYRu0NVyrOJQkSSROSoiKScs3+giGwTkR0i8kRlZRhjUo0x9wDDgV6ejFepmnRp4zD+e19/4huGMPHtNby7co+VLEoXP/rvZHjrZn3WQtnO0zWKt4CBZXeIiBN4FRgEdABGiUgHEeksIp+e82rkes9vgBXAVx6OV6ka1SQiiPmT+3Ftu0Y88/Fm/vLJFmv4LMDA5+HGv+uzFsp2YjzcaSYi8cCnxphOru1+wJ+MMTe6tp8EMMY8X4WyPjPG3FzBsUQgESAuLq7n3r173RK/UjWhuMTw3GepJH2/m2vbNeKlkd0IC/L/5YQtH0Pqp1azVECwfYEqnyYiycaYX7Xc2NFHEQuUnUIz3bWvXCJytYi8LCLTgEUVnWeMmW6M6WWM6RUdHe2+aJWqAU6H8OzgDjx3aye+3X6MYa+vtEZElTq9D07vBYfWLlTNsyNRlDdYvMJqjTHmG2PMg8aYycaYVystWCcFVLXcmL6X8M5dfTiUkcuQV7/nx10nrAOX/Q7uXGT1XxTkwE8f6BBaVWPsSBTpQPMy282Ag+4oWCcFVL6gf+uGfHR/fyKD/Rkz80dm/+hqRi3tq0h+CxZMhCMpFZahlDvZkSjWAG1EpIWIBAAjgYXuKFhrFMpXtIwO5b/39ad/64Y89d8Unv7oJwqKSqyDfSfD+E+gSWdr+8wx+wJVdYKnh8fOAVYCbUUkXUQmGmOKgAeAxUAqMN8Ys9kdn6c1CuVLIur5k3RnbyZf2ZJZq/YxZuYqjmblWetxt7jSOuloKrzUFTbNtzdY5dM8PuqpJonIYGBw69atJ6WlpdkdjlJu8/GGAzy+YBOR9QJ4fWwPusfVtw7kn4Gv/wpXPAyhjewNUtV6FY168qlEUapXr15m7dq1doehlFttPpjB5HeTOZqZzx9/04HRfeKQshMJGgMfToLmfaHPJPsCVbWWNw2PVUpdgI4xEXzywOX0axXFU/9N4eH3N5JbUPzLCUV5kJ8FhbrEqnIvn6pRaNOTqgtKSgz/+XoHL361nbaNw3h1TA9aRYdaB42xXg4H7P0Bjm2FnhN0CnNVJXWiRqGd2aoucDiEh65vw1sT+nAkM4/B/1nBR+sPWAdFrCQBsGE2fP8yFObaF6zyCT6VKJSqS666NJpFD11Bp5gIfj9vA499sJGcgjITCA7+D0z43Jryo7gINr1vTWWuVDX5VKLQ5yhUXdM0oh7vTerLA9e05v3kdG5+eQUb95+2DjocEN7U+jn1Y/jwbti1zL5gVa3lU30UpXTUk6qLVu48wcPzN3AkK5/fX9eGe69uhZ/T9V3QGNj9LbS4ymqeOr0PIppr34U6S53oo1CqLuvXKorPf38lt3Rpyr+/3M7t01ay+3i2dVAEWl5t/Zt1BKZeAd+cd8JmpQAfSxTa9KTquoh6/rw0sjsvj+rOrmPZ3PTSd7y7cg9ntRyERMPl/wNdRljbGemw70edZFBVSJuelPJRhzPyeGzBJpZvP0ZCywZMGdqF+IYhvz7xy2dh5WvwwGpo0LLmA1VeQ5uelKpjmkQE8faE3jw/tDObD2Zy44vLmfrtTgqLzxn5dNXjMGHRL0niyJaaD1Z5NU0USvkwEWFUnziW/r+ruOrSaKZ8vpVbXl7xyzoXAAEh0LyP9fPB9TC1P6yfbU/AyitpolCqDmgcHsS0cT2ZNq4nZ/KLGDF9Fb+fu57DGXnnnNgZBvwF2rlWHD6xEzLdslyMqsV8qo9Cp/BQ6vxyC4p5/ZsdTP12Fw4HTLqiJZOvakVoYDnLrL47FI5tg99vsqY3Vz5NZ49VSp1l/8kc/rl4G59sPEjD0AAevqEtw3s1x+ko82zFqb1wYge0vs4aFbXiBeh8O0TG2Re48hjtzFZKnaV5g2D+M6o7H93fnxYNQ3jyw5/4zSsrWLvn5C8n1b/EShJg1SyWPQ87XU93++CXTFU+rVEopTDG8MmmQzy/KJVDGXkM7NiEB69rQ4eY8LNPzEiHkEbgF2Ctqrf+XRj2FoRE2RK3cq+KahTlNEoqpeoaEeE3XWO4vn0jpi/fxRvf7eaLzYe5sWNjfndtGzrFumZkjmhW9l3g8IPgBtbmnhVWk5Q2S/kcrVEopX4lI6eQpO93k/T9brLyiriiTUMmX9mK/q2jzl5Vr5Qx8HI3iLwExi+09hXkWDPXqlpDO7OVUtWWmVfIez/uI2nFbo5m5dOuSRjjL4tnSLcYggPOaZA4vc9aYa9xR2st73+3gwF/gt532xK7qr46kSh0eKxSnpFfVMxH6w/w1g97ST2USViQH7f1aMbwXs1/3Y8BkHMSfngZ2t0CzXrB0a0wfxwMeQ2a9675C1BVUicSRSmtUSjlGcYY1u49xTsr97I45TAFxSV0jo3gth6x3NSlKY3Cgsp/48H1sPRPVqKIiIW0pbBxDgz6B4Q0rNFrUBXTRKGUcqtT2QV8vOEA89ems+VQJg6BhJZR3NylKQM6NK44aQCsnwUr/g/uWwVOf9jwnvW8xrXP6BoZNtJEoZTymO1Hsvh040E+3XSIXcezEYEecfW5vn1jrmjTkA5Nw3E4zkkAxvySFBY9BulrINH1jMbXz4F/EFzx8K/PVR6jiUIp5XHGGLYdyWLJ5iMs2XKYlAOZANQP9iehZRSdYiPoGBNOx5gIosMCz35zSYm1fCvAgknWlCG3TrW2Zw6A2B5WUxXAnu+hQQsIj6mhK6sb9DkKpZTHiQjtmoTTrkk4D17XhiOZefyw8zgr0k6wZs9JPk85/PO5bRqFcnmbhvRv1ZCOseE0CQ/i5zrDbTPOLviSyyCqlfVzSQnMvh16jLMShzGw6BFrIsNW11rnFOWD3zmJSF0wrVEopWpMZl4hWw5msmH/ab7fcZzVu0+SX2StjxEW5MeljcNo3zSMDk0jaN80jFaNQgkP8j+7kJISOLAWgiIgui3knoZXelvNVAn3QO4p+EcLGPIKdB9rzVc141q46Z/Q6TZrCG/aErikP4Q1seG/gvfSpiellNfJKyxmU3oG2w5nsv3IGbYdziL1UCZZ+UU/nxMVEkCLhiE0q1+PmEjrFRtZj9j61r8hpbPeljZd5ZyE1dOh5TUQ19dKHF8+Cz3GW0N105Nh5rUwcg60uwkOboAPJ8GQV611OY6mWu/v94BVizlzDI6kWO8NDIPCPCvZ1KsPTj+f6j+p1U1PIhICLAf+aIz51O54lFLuEeTvpE+LBvRp0eDnfcYY0k/lknook13Hs9lzPJtdx7NZu/cUhzcdoqjk7C+3YYF+RIcH0igskKjQQBoEB1A/ZCjhe/0IObyP4AAn4W2foSGBRJ3OpUFUe4Lu/eGX6UgcfhDdzvrDD5B1GLZ8DD0nWNt7v4f3x8M930OTTpC60Eosv1tnJZIN78HCB+Chjdb0JetnwfJ/WasGhsdA2pfw0wdWInL6QeqnVvmDX7KeXD+yBU7utJ45EYGTu601QOL7W5+/bxWc3AXdRnv6dlTIo4lCRJKAW4CjxphOZfYPBF4CnMBMY8yU8xT1ODDfY4EqpbyGiNC8QTDNG/x6+o/iEsPRrDwOns4l/VQuB07ncjQzn6NZeRzNzCf1YCYncwrIyC2sdHLbAKeDsKCDBAc6CXA68HdOImjeUcLrnSKiXiThbT6mXrKTIP9tRNCMxt1nkJnmxLlvH/VzmxLT9WmO7i2CI0eIyGtKkw6TOXioCDl1kvDCRjSp34nj2YHUK8kl5PBuQvd8z+FjpyAwhPCj+whOTya7yEmgFBO4cS6y6jV49rgV3JqZkPwW/OGAtZ2yAH56/5dEsfgpyD4OQ6dZ2wt/BwXZMCzJfTfhHB5tehKRK4EzwDuliUJEnMB2YACQDqwBRmEljefPKeIuoAvQEAgCjlelRqFNT0rVbcUlhuyCIrLzi8jOLyYjt4ATZwo4kV3AyewCsvKKyMorJDu/iMJiQ0FxCXmFxWTmFpKRW0hWXhG5hcXkFRZT4uHW+SgyaOrMIE3icTqEeDlMlDlNMu0oMYaQkiykpIjjJoJAPwf3+H1CSznEPwN/h79TGF30EZEmg+lBVg3oheFd6dIs8oJisaXpyRizXETiz9ndB9hhjNnlCmwuMMQY8zxW7eMsInINEAJ0AHJFZJExpqSc8xKBRIC4OJ29Uqm6zOkQwoP8f90RXk3GWEmkoMj1Ki6hsMhQUFxMflEJJSVQWFJCUbGhuMRQYgyFrvNLE40x4BBBBAxWEisqLqGg2JBXaJVTWFxC/5LSMuIQgbaAwyH4OeTnxaQKikrIKLqfVUXF9Cu2PmtD8R0IQltXzPX83b8SoR19FLHA/jLb6UDfik42xjwFICJ3YtUofpUkXOdNB6aDVaNwV7BKqbpLRAj0cxLoV7eXgbUjUZQ3POC8f9iNMW+dt+BfJgW8gLCUUkqVx46lUNOB5mW2mwEH3VGwMeYTY0xiRESEO4pTSimFPYliDdBGRFqISAAwEljojoJFZLCITM/IyHBHcUoppfBwohCROcBKoK2IpIvIRGNMEfAAsBhIBeYbYza74/O0RqGUUu7n6VFPoyrYvwhY5O7P0z4KpZRyPzuanjxGaxRKKeV+PpUolFJKuZ9PJQrtzFZKKffzydljReQYsLcab2kIHPdQON6sLl53XbxmqJvXrddcfZcYY6LP3emTiaK6RGRtefOb+Lq6eN118Zqhbl63XrP7+FTTk1JKKffTRKGUUqpSmigs0+0OwCZ18brr4jVD3bxuvWY30T4KpZRSldIahVJKqUppolBKKVWpOp8oRGSgiGwTkR0i8oTd8XiCiDQXkWUikioim0XkIdf+BiLypYikuf6tb3es7iYiThFZLyKfurZbiMiPrmue55rB2KeISKSIfCAiW133vJ+v32sR+R/X73aKiMwRkSBfvNcikiQiR0Ukpcy+cu+tWF52/W3bJCI9LvRz63SicK3f/SowCGup1VEi0sHeqDyiCHjYGNMeSADud13nE8BXxpg2wFeubV/zENYsxaX+Afyf65pPARNticqzXgK+MMa0A7piXb/P3msRiQUeBHoZYzoBTqzlC3zxXr8FDDxnX0X3dhDQxvVKBF6/0A+t04mCMut3G2MKgLnAEJtjcjtjzCFjzDrXz1lYfzhisa71bddpbwO/tSdCzxCRZsDNwEzXtgDXAh+4TvHFaw4HrgTeADDGFBhjTuPj9xprJux6IuIHBAOH8MF7bYxZDpw8Z3dF93YI8I6xrAIiRaTphXxuXU8U5a3fHWtTLDVCROKB7sCPQGNjzCGwkgnQyL7IPOJF4DGgdJ31KOC0a00U8M373RI4BrzpanKbKSIh+PC9NsYcAP4F7MNKEBlAMr5/r0tVdG/d9vetrieKC1q/u7YSkVBgAfB7Y0ym3fF4kojcAhw1xiSX3V3Oqb52v/2AHsDrxpjuQDY+1MxUHleb/BCgBRADhGA1u5zL1+71+bjt972uJwqPrd/tbUTEHytJzDbGfOjafaS0Kur696hd8XlAf+A3IrIHq0nxWqwaRqSreQJ8836nA+nGmB9d2x9gJQ5fvtfXA7uNMceMMYXAh8Bl+P69LlXRvXXb37e6nig8tn63N3G1zb8BpBpjXihzaCEw3vXzeODjmo7NU4wxTxpjmhlj4rHu69fGmDHAMmCY6zSfumYAY8xhYL+ItHXtug7Ygg/fa6wmpwQRCXb9rpdes0/f6zIqurcLgTtco58SgIzSJqrqqvNPZovITVjfNJ1AkjHmOZtDcjsRuRz4DviJX9rr/4DVTzEfiMP6n+12Y8y5HWW1nohcDTxijLlFRFpi1TAaAOuBscaYfDvjczcR6YbVgR8A7AImYH0p9Nl7LSJ/BkZgjfBbD9yN1R7vU/daROYAV2NNJ34E+CPwEeXcW1fSfAVrlFQOMMEYs/aCPreuJwqllFKVq+tNT0oppc5DE4VSSqlKaaJQSilVKU0USimlKqWJQimlVKU0USjlBUTk6tIZbpXyNpoolFJKVUoThVLVICJjRWS1iGwQkWmu9S7OiMi/RWSdiHwlItGuc7uJyCrXWgD/LbNOQGsRWSoiG13vaeUqPrTMOhKzXQ9MISJTRGSLq5x/2XTpqg7TRKFUFYlIe6ynf/sbY7oBxcAYrEno1hljegDfYj0tC/AO8LgxpgvWU/Gl+2cDrxpjumLNSVQ6rUJ34PdYa6O0BPqLSAPgVqCjq5y/efYqlfo1TRRKVd11QE9gjYhscG23xJoWZZ7rnFnA5SISAUQaY7517X8buFJEwoBYY8x/AYwxecaYHNc5q40x6caYEmADEA9kAnnATBEZijUVg1I1ShOFUlUnwNvGmG6uV1tjzJ/KOa+yeXHKm/q5VNl5iIoBP9d6Cn2wZv79LfBFNWNW6qJpolCq6r4CholII/h5reJLsP4/Kp2ldDSwwhiTAZwSkStc+8cB37rWAUkXkd+6yggUkeCKPtC1hkiEMWYRVrNUN09cmFKV8Tv/KUopAGPMFhF5GlgiIg6gELgfa3GgjiKSjLW62gjXW8YDU12JoHQWV7CSxjQR+YurjNsr+dgw4GMRCcKqjfyPmy9LqfPS2WOVukgicsYYE2p3HEp5ijY9KaWUqpTWKJRSSlVKaxRKKaUqpYlCKaVUpTRRKKWUqpQmCqWUUpXSRKGUUqpS/x8QDdUuG4r7bgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_and_plot(poly_features[:n_train, :], poly_features[n_train:, :], labels[:n_train], labels[n_train:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_pu3gps2",
    "id": "F5C7FE5EF88D44FE85E0B651B3B09959",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 线性函数拟合（欠拟合）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "graffitiCellId": "id_07e409i",
    "id": "33AD626DA0B94DB7A28D47697312B45D",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "final epoch: train loss 101.62761688232422 test loss 77.33094024658203\n",
      "weight: tensor([[15.2018]]) \n",
      "bias: tensor([0.1108])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEGCAYAAACtqQjWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd5xddbnv8c8zPVMyyUwaKTChhRJ64IIgBxQwIB7wiKCIDQW9x4IFVLzqORzLBQUVFJCuXDBIryGEQEIQA6SQQCpJIGXSJjOT6Zn+3D/W3tkzySSZsvesmb2/79drv7J/qz6rZD/z+/1WMXdHREQkHtLCDkBERJKHkoqIiMSNkoqIiMSNkoqIiMSNkoqIiMRNRtgBhG3EiBFeUlISdhgiIoPKwoULy9195O7DUz6plJSUsGDBgrDDEBEZVMxsfVfD1fwlIiJxo6QiIiJxo6QiIiJxk/J9KiIiPdXS0kJpaSmNjY1hh5JwOTk5jB8/nszMzG5Nr6QiItJDpaWlFBQUUFJSgpmFHU7CuDsVFRWUlpYyceLEbs2j5i8RkR5qbGykuLg4qRMKgJlRXFzcoxqZkoqISC8ke0KJ6ul2Kqn00iNvb+CJhaVhhyEiMqAoqfTSk4s28djCjWGHISIpqKqqijvuuKPH811wwQVUVVUlIKIYJZVeKsrLorK+OewwRCQF7S2ptLW17XO+6dOnM2zYsESFBSip9FpxfhYVdUoqItL/fvKTn7B27VqOP/54Tj75ZM4++2wuv/xyjjnmGAAuvvhiTjrpJI4++mjuvvvuXfOVlJRQXl7OunXrOPLII7nqqqs4+uijOe+889i5c2dcYtMlxb1UnJfFjoZm2tqd9LTU6LATkT3d8Nwylm+uiesyjxo7lP/61NF7HX/jjTeydOlSFi9ezJw5c/jkJz/J0qVLd132e//991NUVMTOnTs5+eST+cxnPkNxcXGnZaxevZpp06Zxzz33cOmll/LEE09wxRVX9Dl21VR6qTg/m3aHqgbVVkQkXKecckqn+0huu+02jjvuOE499VQ2btzI6tWr95hn4sSJHH/88QCcdNJJrFu3Li6xqKbSS0V5WQBU1jdTnJ8dcjQiEpZ91Sj6S15e3q7vc+bMYdasWcybN4/c3FzOOuusLu8zyc6O/W6lp6fHrflLNZVeKs4Pkkq5+lVEpJ8VFBRQW1vb5bjq6mqGDx9Obm4uK1eu5M033+zX2FRT6aXivCDL6wowEelvxcXFnH766UyePJkhQ4YwevToXeOmTp3KX/7yF4499lgmTZrEqaee2q+xKan0UrSmUlHfFHIkIpKK/v73v3c5PDs7mxdffLHLcdF+kxEjRrB06dJdw6+99tq4xaXmr14anpuFGbqsWESkAyWVXkpPM4bnZqmmIiLSgZJKH+iuehGRzpRU+qA4L0tXf4mIdKCk0gfBo1rU/CUiEqWk0gfFedlq/hIR6UBJpQ+K8rLY0dBCa1t72KGISArp7aPvAf74xz/S0NAQ54hilFT6YETkXpUdDS0hRyIiqWQgJxXd/NgHRZG76ivqmxhZoOd/iUj/6Pjo+3PPPZdRo0bx6KOP0tTUxKc//WluuOEG6uvrufTSSyktLaWtrY2f//znbNu2jc2bN3P22WczYsQIZs+eHffYVFPpg+hd9ZW6AkwktT3wSXjn4eB7W0tQXvKPoNzcEJSXPhGUG6uD8vJng3J9RVBeFbkLvnbbfld34403csghh7B48WLOPfdcVq9ezdtvv83ixYtZuHAhc+fOZcaMGYwdO5YlS5awdOlSpk6dyne/+13Gjh3L7NmzE5JQQEmlT4ojTyouV2e9iIRk5syZzJw5kxNOOIETTzyRlStXsnr1ao455hhmzZrFj3/8Y15//XUKCwv7JR41f/VB9JH3lbqsWCS1ffWF2Pf0zM7lrNzO5ZzCzuW84s7lgtjDIbvD3bn++uv5xje+sce4hQsXMn36dK6//nrOO+88fvGLX/Ro2b2hmkofDBuSSZpBhWoqItKPOj76/hOf+AT3338/dXV1AGzatImysjI2b95Mbm4uV1xxBddeey2LFi3aY95EUE2lD9LSjKK8LCUVEelXHR99f/7553P55Zdz2mmnAZCfn89DDz3EmjVruO6660hLSyMzM5M777wTgKuvvprzzz+fAw44ICH9KubucV/oYDJlyhRfsGBBr+c/7w+vMXFEHnd9cUocoxKRgWzFihUceeSRYYfRb7raXjNb6O57/PCp+auPdFe9iEiMkkofFeVn6Z0qIiIRSip9NEJ9KiIpKVW6Dnq6nUoqfVScn031zhZa9PwvkZSRk5NDRUVF0icWd6eiooKcnJxuz6Orv/qoKHID5I76ZkYN7f6OF5HBa/z48ZSWlrJ9+/awQ0m4nJwcxo8f3+3plVT6KPpQyfI6JRWRVJGZmcnEiRPDDmNAUvNXH3V8qKSISKpLyqRiZheb2T1m9oyZnZfIde16qKQ660VEEpdUzGyCmc02sxVmtszMrunDsu43szIzW9rFuKlmtsrM1pjZTwDc/Wl3vwr4CnBZrzeiG3Y9VFKXFYuIJLSm0gr80N2PBE4FvmVmR3WcwMxGmVnBbsMO7WJZfwWm7j7QzNKB24HzgaOAz++2jp9FxifM0JxMMtKMSjV/iYgkLqm4+xZ3XxT5XgusAMbtNtm/Ac+YWQ6AmV0F3NbFsuYClV2s5hRgjbt/4O7NwCPARRa4CXgxGsPuzOxTZnZ3dXV17zawvhyqNpCWZgzP0w2QIiLQT30qZlYCnAC81XG4uz8GzAAeMbMvAFcCl/Zg0eOAjR3KpZFh3wHOAS4xs292NaO7P+fuV/f6HQNPfA0e/xoQNIHpBkgRkX64pNjM8oEngO+5e83u4939t2b2CHAncIi71/Vk8V0Mc3e/jS5qPHF1+jXgwQ2PxflZlOudKiIiia2pmFkmQUJ52N2f3Ms0HwUmA08B/9XDVZQCEzqUxwObexFqzx3yMTj0HABGFeSwvVZJRUQkkVd/GXAfsMLdf7+XaU4A7gEuAr4KFJnZr3qwmvnAYWY20cyygM8Bz/Yt8m7aWQVb3oX2NkYPzaGspinpH9kgIrI/iaypnA58EfiYmS2OfC7YbZpc4LPuvtbd24EvA+t3X5CZTQPmAZPMrNTMvgbg7q3At4GXCC4EeNTdlyVukzp47zG466PQUMnoodk0t7XrXhURSXkJ61Nx93/SdZ9Hx2ne2K3cQlBz2X26z+9jGdOB6b0Ms/cO/Thc9hBk5TFmaCsAW2sad723XkQkFenZX71VdHDwAUYXBv0pZTVNHD02zKBERMKVlI9p6RetTbBpEdSVMTryIMmtNY0hByUiEi4lld6qL4d7zoaVLzCqIBsz2FqtpCIiqU3NX72VPwo+Nw0OOJbM9DSK87LZppqKiKQ4JZXeSs+EI2IXs40pVFIREVHzV19sfS/4AKMLcthaoxsgRSS1Kan0xTPfhlk3ADC6MEc1FRFJeWr+6otP3gKZuQCMGZpDZX0zTa1tZGekhxyYiEg4lFT6YvyUXV/HRC4rLqtpYkJRblgRiYiESs1ffVH5IayaAcCoocGd9GoCE5FUpqTSF0sfh2mXQWszYwp1A6SIiJq/+uK4zwePv09L39X8pRsgRSSVKan0ReH44AMUDkkjKyNNzV8iktLU/NUXjdWw4jmo2YyZMWZoDtt0r4qIpDAllb6o2Qz/uAI2vAkEV4CpT0VEUpmav/pi+ET4xuswvAQIboB8t7Qq3JhEREKkpNIXmTlwwLG7iqMLstla3Yi7E7xNWUQktaj5q69WTof18wAYU5hDU2s7NTtbQw5KRCQcSip99dJPYf69AHpZl4ikPDV/9dUXn4TsQoBON0BOGlMQZlQiIqFQUumryHvqIXj8PcA23QApIilKzV99VboA3nkI0PO/RESUVPpq2VPwwrUA5GSmMzw3U30qIpKylFT66qM/hO+9B+5A0FmvmoqIpCr1qfRVblGn4mjdVS8iKUw1lb6q3gTz7gge2QKMHZbD5iolFRFJTUoqfVW9EV66HspWADChKJfK+mbqmnQDpIikHiWVvhp7IvzoQzj4bAAOjLxKeGNlQ5hRiYiEQkmlrzKygn6VtGBXRpPKBiUVEUlBSirxMO8OWPsqoJqKiKQ2JZV4eP1mWPE8AIVDMinIyVBSEZGUpEuK4+GaJZCVD4CZcWBRrpq/RCQlqaYSD9kF0OH9KROGK6mISGpSUomHlS/AG7fuKh5YnMvGHTtpb/cQgxIR6X9KKvGw5hV48y+7ihOKcmlubaestinEoERE+p+SSjycfxP8YPmuoi4rFpFUpaQSD+mZnfpUlFREJFUpqcRD5Qcw46dQsRaAccOGYKakIiKpR0klHnZWwcIHoGo9AFkZaYwtHKJ7VUQk5eg+lXgYewL8dHPny4qLhqimIiIpRzWVeDDrlFAA3QApIilJSSVeZv8GFv5tV/HAoly21zaxs7ktxKBERPqXkkq8fDAHNr+zqzgh+mDJHaqtiEjqUJ9KvHxtZqfirsuKKxo4fHRBGBGJiPQ71VQSRPeqiEgqUlKJl1UvwuNXggfP+yrKyyIvK11JRURSipJKvNRugS1LoLkOCB6BP6EoV/eqiEhKUZ9KvEy5Mvh0cGBRLh+W14cUkIhI/1NNJYEOKs5lfWUDbXoEvoikCCWVeKkvh0e/FDwGP+Lw0QU0t7azvkK1FRFJDUoq8ZKRDWUrYOeOXYMmjQkuJV61tTasqERE+pWSSrxkF8C358Mxl+wadNioAsxgpZKKiKSIbiUVM7vGzIZa4D4zW2Rm5yU6uMFuSFY6JcV5qqmISMrobk3lSnevAc4DRgJfBW5MWFSD1Su/hOk/6jRo0ugC3t+mpCIiqaG7SSX6CN4LgAfcfUmHYRLVshOaO3fKTxpTwLqKehpb9GBJEUl+3b1PZaGZzQQmAtebWQHQnriwBqmpv9lj0KQxBbQ7rN5WxzHjC0MISkSk/3S3pvI14CfAye7eAGQSNIHJfkSvAFu5tSbkSEREEq+7SeU0YJW7V5nZFcDPgOrEhTVIfTAH7j0Hqkt3DSopziM7I039KiKSErqbVO4EGszsOOBHwHrgwYRFNVilZ0NWPrQ2xQalGYeNztdlxSKSErqbVFrd3YGLgFvd/VZALwnZ3UGnwZeehuJDOg2eNHqoLisWkZTQ3aRSa2bXA18EXjCzdIJ+FemGSWPyKattYkd9c9ihiIgkVHeTymVAE8H9KluBccDvEhbVYOUO954L//pzp8GTxgwFdGe9iCS/biWVSCJ5GCg0swuBRndXn8ruzKBwPOR0vnT4iMgVYOqsF5Fk1637VMzsUoKayRyCmx7/ZGbXufvjCYxtcPrsA3sMGlWQzbDcTNVURCTpdffmx/9DcI9KGYCZjQRmAUoq3WBmHD66gFW6V0VEklx3+1TSogkloqIH86aW134Hd525x+AjxhTw/rY62vXCLhFJYt1NDDPM7CUz+4qZfQV4AZieuLAGsWEHwgHHBZ32HUweV0hdUysflNeFFJiISOJ1q/nL3a8zs88ApxP0qdzt7k8lNLLB6rjLgs9uTi4pAmD+uh0cOkq3+IhIcupunwru/gTwRAJjSWolxbmMyM9i/rpKPn/KgWGHIyKSEPts/jKzWjOr6eJTa2bqde5K7Va45Qh45+FOg82MKQcVsWDdjr3MKCIy+O0zqbh7gbsP7eJT4O5D+yvIQSW3GA75OAybsMeoKSXD2VDZwLaaxhACExFJPF3BFW/pmXDx7TBxzyvAov0qqq2ISLJSUkmU9j3fYXbU2KEMyUxn/rrKEAISEUk8JZVEeP4H8OeT9hicmZ7G8ROGsWC9koqIJCcllUSY+FE4/vIuR51cMpzlm2uoa2rt56BERBKv25cUSw8c/em9jppSUkS7w+INVZxx2Ih+DEpEJPFUU0mUthZo3fP9KSccOIw0Q/0qIpKUlFQSoXw1/GoUrHh2j1EFOZkcecBQ9auISFJSUkmEoWPhzOtg5BFdjj65pIhF66toadvzCjERkcFMSSURsvLg7J/CmMldjj714GJ2trSpCUxEko6SSqK0tUB9eZejzjx8BDmZacxYurWfgxIRSSwllUR57Cvw1wu7HJWblcG/HT6Sl5Zt1ftVRCSpKKkkyklfgTO+v9fR508+gG01TSwureq/mEREEkz3qSTKYefuc/TZR4wiM92YsXQrJx44vJ+CEhFJLNVUEqWtFXash6au3/RYOCSTjxwyghlLt+KuJjARSQ5KKomydQnceix8MGevk5w/eQwbKhtYsaW2/+ISEUkgJZVEKT4MPnUrHHDsXic596jRpBnMWLqlHwMTEUkcJZVEyRkadNYP2/urg4vzszllYhEzlunSYhFJDkoqiVSzBba/v89Jph49hve31bF8s97OLCKDn5JKIj3zn/DUN/Y5ycUnjCM/O4PbZ6/pp6BERBJHlxQn0pnXge/7+V7DcrP48kcO4o45a3l/Wy2Hjy7op+BEROJPNZVEOugjUHLGfif7+hkHMyQznT+9qtqKiAxuSiqJ1LIT1s+DurJ9TjY8L4svnVbC8+9uZk2ZLi8WkcFLSSWRajbDA1Nh9cv7nfSqj04kJyOdP6u2IiKDmJJKIg07CL7wOBz+if1OWpyfzRdPO4hnl2xm1VbVVkRkcFJSSaT0jOAZYHndexf9N848mMIhmVz3+BJa9QIvERmElFQSrXwNLH+mW5MW52fzy4sn825pNXfN/SDBgYmIxJ+SSqItmQaPfRVam7o1+YXHjuWCY8Zw66zVagYTkUFHSSXRplwJ33oL0jK7PcsvL5pMQU4G1z62RO+xF5FBRUkl0QrHwYjDIK37uzraDPbepmp+/MS7tOntkCIySCip9IelT8C6N3o0ywXHHMAPzj2cJxdt4von39Vrh0VkUNBjWvrDzF9E7q4/vUezfffjh9Ha1s5tr64hPS2NX188mbQ0S1CQIiJ9p6TSH746HfJH92rW7597OC3tzp1z1rK9tonfXXIsw/Oy4hygiEh8qPmrPww/CDJzejWrmfGjT0ziFxcexdz3tzP11rn8a215nAMUEYkPJZX+UPkhzL0Z6nuXDMyMK8+YyJP/+RHysjP4wr1vcd1jS1i7vS7OgYqI9I2SSn+o3giv/hK2r+zTYiaPK+T575zBladP5Ll3N3PO71/jWw8vYuH6StzVkS8i4bNU/zGaMmWKL1iwILEraW2GtmbIzo/bIsvrmnjgjQ958F/rqW1q5eCReVw6ZQLnHjWaicV56tAXkYQys4XuPmWP4Uoq/ZBUEqi+qZUX3tvCYws2Mn/dDgAKcjI4bvwwjhhTwMSReUwckceE4bmMLMgmJzM95IhFJBkoqexFvyWV9x6Hmk1w+jUJW8W68nreXlfJko1VLCmtYvW2OppaO9+RXzgkkxH5WRTlZTE8N4thuZnkZWeQl5VBbnY62RnpZGWkkZ2eRlqakZFmpKcZmelpZGUYGWlppJmRZkFfT3pa7Lt1qBx1rCeZdV1rSkRdqrtnc+f4uljOfha0l03q9vzdXWY8/nvuL9ZECONnJVH7rysO3W5yjp7/0fCic+0+v5n16P9Exxg6Lmn39ew+vOP/x6PHDiUzvXe9IHtLKkl1SbGZXQx8EhgF3O7uM0MOKWbtq7BlSUKTSsmIPEpGBM1gAO3tzpaaRj7cXs/mqp2U1TZSVttEeV0TlfXNrK9oYElpMw1NbdQ1t4byQyAi4Vnws3MYkZ8d12UO+KRiZvcDFwJl7j65w/CpwK1AOnCvu9/o7k8DT5vZcOBmYOAklU/dCundf/5XPKSlGeOGDWHcsCH7ndbdaWptj3zaaG5tp70dWtvbaWt3WtqclrZ2WtraaXdod6fdHY987/gomU65aS+Jyrtdp+i5/f2913Hd+0qke/sLv7vJd29/Off0L+q+1DTC/EOhO3Hva3/0ZLsTtf/2xbD9V7c9+s9utZLojB2qFb35P9ExBqPDfzcPBuxRa9ltFQU58U8BAz6pAH8F/gw8GB1gZunA7cC5QCkw38yedfflkUl+Fhk/cPRzQukpMyMnMz3S5zKwYxWRgWvAX1Ls7nOByt0GnwKscfcP3L0ZeAS4yAI3AS+6+6K9LdPMrjazBWa2YPv27YkLviN3mP4jWPT/+md9IiIhGPBJZS/GARs7lEsjw74DnANcYmbf3NvM7n63u09x9ykjR45MbKRRZrBpIVSu7Z/1iYiEYDA0f3Wlq5ZMd/fbgNv6O5hu+/qscC7FERHpJ4O1plIKTOhQHg9sDimW7lNCEZEkN1iTynzgMDObaGZZwOeAZ0OOaf9qt8GDF8PK6WFHIiKSEAM+qZjZNGAeMMnMSs3sa+7eCnwbeAlYATzq7svCjLNbhgyHpprgkS0iIklowPepuPvn9zJ8OjC4/uTPyIKrXg07ChGRhBnwNRURERk8lFT62/p/wR8mw9b3wo5ERCTulFT629CxMP5kMO16EUk+A75PJekML4HPPhB2FCIiCaE/l8PS0hh2BCIicaekEoa37oIbD4Tm+rAjERGJKyWVMIw7KXiviu5XEZEkoz6VMIyfEnxERJKMaiphaW+HzYvDfYuSiEicKamEZck0uPvfoGz5/qcVERkklFTCcti58Om7oXB82JGIiMRNyiYVM/uUmd1dXV0dTgD5o+C4yyCnMJz1i4gkQMomFXd/zt2vLiwM8Ue9sRqW/AN27ggvBhGROErZpDIgbH8fnroa1s4OOxIRkbjQJcVhGncifP1VGHtC2JGIiMSFkkqY0tJh/ElhRyEiEjdq/gpbUy288kv4cG7YkYiI9JmSStjSs2Hx32HDW2FHIiLSZ2r+CltGFnx7PmTnhx2JiEifqaYyEEQTSnNDuHGIiPSRkspAseQR+P0RUF8ediQiIr2mpDJQjD0BJl8C7W1hRyIi0mvqUxkoRk6CC38fdhQiIn2imspAU7EWlj8TdhQiIr2ipDLQvHIDvPhjaG0KOxIRkR5T89dAM/UmMIOM7LAjERHpMSWVgWboAbHvDZWQWxReLCIiPaTmr4Fq5s+DN0O2NIYdiYhIt6mmMlAdPhWy8tQMJiKDipLKQFVyevCB4CVeOcOCvhYRkQFMzV8DXV0Z/OVMeP2WsCMREdkvJZWBLm8kHPXvcOjHw45ERGS/UjapmNmnzOzu6urqsEPZNzP4xK9jb4d852Go2RxuTCIie5GyScXdn3P3qwsLC8MOpfvqy4MbI9+4LexIRES6pI76wSRvBFw9B4aODcqbFsL6eXDKVbpKTEQGhJStqQxaIw6FrNzg+6oXYe5vwSKHccsSKF8dXmwikvKUVAazj/0MvvkGpGcG5enXwdP/Ozb+9Vtg0YOx8pYlsGNdrNxY0/kZY22t4N69dbtDU13s5sz29uBhmA2VsWVtnA81W4JyazN8MAeqS4NyUy0seAC2rwrKzfWweBpUfhCLbdlTULUhMn0drHkluBoOYGcVrJoRK9dXBA/irNselKs2wrzboXpTrDz/PqjdFpm+HD54LYgDoK0liDmqvW3f+6K1OdjW6DRNdUFfV3t7LL6KtR3G1wbDouWGymB8VFvrvl97ULe98x8M5Wtg06JYedMiWPdGrLz9fdi2LFau3hTbFwC1W2P7oqvxZSs7r6+xuvNL5Fqbgn0WtbMqOGa7llfa+d1A694IYo7asqTz+jvua/fgvIruy/1xD45HdPro/K3NQbm5IXhddzSehsrgXKvaGJTrtgf/T6Ln5s4dwbkWPZfrymDZ08E5BsE5+fY9sXNt+/sw56bY9mxbBq/+Kra+ms2weha07Izsm02w+uVYfBVrYfmzsf9LtduC/RPdnupNQYtEdB+Vr4b3X+qwb/8ZnOtRG94K+l6jVr4Ab9waK6+fF6wvQZRUBrthE2LfL7oDpt4YK69+ufMPzaNfCk72qDtOhRd+GCvfcji88INY+bYT4NVfx8q/GQ+zboiV/+84+Ocfgu/trfCnE2HB/UG5dSfcdw4sfTwoN9fBgxcFJzgESeT578H6fwXlujJ4+pvBfwgIfvQe+wpsfDso71gHD/0HbHgzKFeuhWmXxX5YK1YH27ftvcj0H8JLP40l0bLlwbZFfzg2vAkP/jtUfhiUVz4PvyyO/ZAueQT+pyj2w794Gvzu0NgPxfx74LcTgx9bgAX3we+PDLYb4O27g/0RTRT/+hPcdFDsh+H1W+DOj8T25cs/h5smxsrP/wBunhQrz/wZ/L//iJXn/Aae+Hqs/PotMP3aztM//Z+x8jPfCvZn1ONXBp+oR78Iz34nVn7y68Eyou77BDz1jVj59v/Vefl3fgReuj5WvvssmP2bWHna52H+vR2Wdx7M+3Pw3R1+ORJe+11QbmuGX4+OXUbfVAe/Gg3z7gjKO6vgd4cFxwSgZhP8aiQs+ltQrvwgmH/ZU0G5eiPcfx58+Fps+qe/GfxwQ5Aknv0ObFselMtWBOfa1ncj5eXw2JehPPIHUNnKYF9XrQ/K5auC41Ef+QNn+0qYezM0RJLQmlfg4c9AfSQJrX4JHr4kNv79l4L9Hz133n0E7joTWiNJZvHD8MDU2Ln07j9g2udi+3L1yzDrv2Pn1nuPwYwOx2L1THjrrlh54V87H9t4c/eU/px00kme1NrbY98/fN198+JYef797qtfjpXfuM191Uux8vQfuy99Mlae9T/uq2fFyv+81X39vNh6Fj/ivnVZUG5rdX//ZffKD4Nya7P7h/90r94cG1+92b2pPja+Yq37zqqg3NLovm25+87qoNxYG6yrviIoN9W5ly5wb9gRK295L5guuryGSvfWltjyara6tzQF5fqKYH9Ep9+6zH32je512yPlpcH2Rte/7g33574XW//mxe7z7ojFv+XdYH+2NkfK77kvnhZsp7v7hrfd3/xLbN9tWuT+3uOx8vsz3f/5x1h5xQvuc2+JlTe85b5qRqy85T339W/GyhVrg5g7Lr/j+LWzg+Oxa30v71leOydWXvdGsIyodx52X/lirPz2vUGMUYseCtYR9e5jwTZHrZ8XxOgenCsrX+x8rrz669i52Nbm/trvYudWS6P7zJ8Hx8vdvbHG/dlrYtM31bvP+W3s3N5Z7T735tj+aKoLtq9ma2x5FWtjx4teJxUAAAa/SURBVK6lyX3H+lh5Z3Ww7ui51VgTxLpr+kb32rLYudXeHsTc8f9aR3XlwbFoaQzKNVvdN86PlesrguMZPVfK17gvfy62/Iq17mteCdbh7l5VGpz7UU31sWVF468t6xxDdN5oPDvWdx1rDwALvIvfVPPuNnckqSlTpviCBQvCDkNEZFAxs4XuPmX34Wr+EhGRuFFSERGRuFFSERGRuFFSERGRuFFSERGRuFFSERGRuFFSERGRuFFSERGRuEn5mx/NbDuwvgezjADK9ztVcknFbYbU3O5U3GZIze3u6zYf5O4jdx+Y8kmlp8xsQVd3kSazVNxmSM3tTsVthtTc7kRts5q/REQkbpRUREQkbpRUeu7usAMIQSpuM6TmdqfiNkNqbndCtll9KiIiEjeqqYiISNwoqYiISNwoqXSTmU01s1VmtsbMfhJ2PIliZhPMbLaZrTCzZWZ2TWR4kZm9bGarI/8ODzvWeDOzdDN7x8yej5QnmtlbkW3+h5llhR1jvJnZMDN73MxWRo75acl+rM3s+5Fze6mZTTOznGQ81mZ2v5mVmdnSDsO6PLYWuC3y+/aumZ3Y2/UqqXSDmaUDtwPnA0cBnzezo8KNKmFagR+6+5HAqcC3Itv6E+AVdz8MeCVSTjbXACs6lG8C/hDZ5h3A10KJKrFuBWa4+xHAcQTbn7TH2szGAd8Fprj7ZCAd+BzJeaz/Ckzdbdjeju35wGGRz9XAnb1dqZJK95wCrHH3D9y9GXgEuCjkmBLC3be4+6LI91qCH5lxBNv7t8hkfwMuDifCxDCz8cAngXsjZQM+BjwemSQZt3kocCZwH4C7N7t7FUl+rIEMYIiZZQC5wBaS8Fi7+1ygcrfBezu2FwEPRl4//yYwzMwO6M16lVS6ZxywsUO5NDIsqZlZCXAC8BYw2t23QJB4gFHhRZYQfwR+BLRHysVAlbu3RsrJeMwPBrYDD0Sa/e41szyS+Fi7+ybgZmADQTKpBhaS/Mc6am/HNm6/cUoq3WNdDEvqa7HNLB94Avieu9eEHU8imdmFQJm7L+w4uItJk+2YZwAnAne6+wlAPUnU1NWVSB/CRcBEYCyQR9D0s7tkO9b7E7fzXUmle0qBCR3K44HNIcWScGaWSZBQHnb3JyODt0Wrw5F/y8KKLwFOB/7dzNYRNG1+jKDmMizSRALJecxLgVJ3fytSfpwgySTzsT4H+NDdt7t7C/Ak8BGS/1hH7e3Yxu03Tkmle+YDh0WuEMki6Nh7NuSYEiLSl3AfsMLdf99h1LPAlyPfvww809+xJYq7X+/u4929hODYvuruXwBmA5dEJkuqbQZw963ARjObFBn0cWA5SXysCZq9TjWz3Mi5Ht3mpD7WHezt2D4LfClyFdipQHW0mayndEd9N5nZBQR/vaYD97v7r0MOKSHM7AzgdeA9Yv0LPyXoV3kUOJDgP+Zn3X33TsBBz8zOAq519wvN7GCCmksR8A5whbs3hRlfvJnZ8QQXJ2QBHwBfJfhjM2mPtZndAFxGcKXjO8DXCfoPkupYm9k04CyCR9xvA/4LeJoujm0kwf6Z4GqxBuCr7r6gV+tVUhERkXhR85eIiMSNkoqIiMSNkoqIiMSNkoqIiMSNkoqIiMSNkorIIGNmZ0WfpCwy0CipiIhI3CipiCSImV1hZm+b2WIzuyvyvpY6M7vFzBaZ2StmNjIy7fFm9mbkXRZPdXjPxaFmNsvMlkTmOSSy+PwO70F5OHLzGmZ2o5ktjyzn5pA2XVKYkopIApjZkQR3bZ/u7scDbcAXCB5guMjdTwReI7jLGeBB4MfufizB0wyiwx8Gbnf34wieURV9dMYJwPcI3u9zMHC6mRUBnwaOjiznV4ndSpE9KamIJMbHgZOA+Wa2OFI+mODRN/+ITPMQcIaZFQLD3P21yPC/AWeaWQEwzt2fAnD3RndviEzztruXuns7sBgoAWqARuBeM/sPgsdtiPQrJRWRxDDgb+5+fOQzyd3/u4vp9vWcpK4eRx7V8blUbUBG5H0gpxA8YfpiYEYPYxbpMyUVkcR4BbjEzEbBrneDH0Twfy76NNzLgX+6ezWww8w+Ghn+ReC1yHtsSs3s4sgyss0sd28rjLwDp9DdpxM0jR2fiA0T2ZeM/U8iIj3l7svN7GfATDNLA1qAbxG8COtoM1tI8NbByyKzfBn4SyRpRJ8WDEGCucvM/ieyjM/uY7UFwDNmlkNQy/l+nDdLZL/0lGKRfmRmde6eH3YcIomi5i8REYkb1VRERCRuVFMREZG4UVIREZG4UVIREZG4UVIREZG4UVIREZG4+f84p/fN6PIb6AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_and_plot(features[:n_train, :], features[n_train:, :], labels[:n_train], labels[n_train:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_hmoo3h2",
    "id": "BC253732133341DC8F028FB6834C68F0",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 训练样本不足（过拟合）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "graffitiCellId": "id_h1gobje",
    "id": "AB13F65A70A9484788F8004E427EC290",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "final epoch: train loss 0.5679418444633484 test loss 262.86151123046875\n",
      "weight: tensor([[0.2020, 0.1659, 0.1278]]) \n",
      "bias: tensor([4.0434])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxV9Z3/8deXEAhLWBMC2QiIgoCsYasbUzdQFFdcQB270P7amXEWO9Xp2E4706ndrdPFWmtdQRFBQVFxAUElQMImexAISSAkbCGE7Pn8/jh3SRRkMZebe+/7+XjwkM+52+dw7rlvz/Y9zswQEREBaBPuBkREpPVQKIiISIBCQUREAhQKIiISoFAQEZGAtuFu4MtISkqyrKyscLchIhJR8vLyDphZ8okei+hQyMrKIjc3N9xtiIhEFOdcwcke0+4jEREJUCiIiEiAQkFERAIUCiIiEhCRoeCcu94590R5eXm4WxERiSoRGQpmttDMZnbt2jXcrYiIRJWIDAUREQkNhYJIpGuo9/741R6H+ppgXV3uTfM7VgY1FcG6vNh7DoAZHNoFVYe9urERDuTD8UO+ugH2b4LKg77ProO966DygFfX10BRrvcZAHXVULAiWNdWwq5lcKzUq2sq4NP3g49XHYHti4P18UOwdVHw/Y+VweYFwc+vKIGN84L9lRfDhpeD/R8ugPUveu8LcGgnrH0eqo969YF8yHsGao55dekWWP3X4L9XySew6i/Bf8+9ayHncW++wZvXFX/w/p0A9uTAx78P/tvu/rB5/ekS7/l++e82r7e9BSv+GKy3LPQ+z2/TfFj5Z0JJoRCNzLwvqf+L2tjorYz+L3Jjg7dS1VV7dX0tlBcFV4S6am9l8a8otZXeyuH/Iakuh8JVwRXr+CHY+UHwh+VYqffl9tflxd6X2V8f2gXrZgffr2w75D4V/LySjd6KUVvp1cV5sPzXwX735MD7P/X6Bti5FBb/Z3B+t78Ni/49+O+x6VV4/V+C9fqXYME/Buu8Z+DV7wTrnMdh3reC9fLfwCvfDNbv/xRe+UawfvsHMPdrwfr1f4WX/z5Yz/82vHxfsH7p7ub1czc3f/1Tk5rXj18Kc78erB8bCfNmBuvfDYOF9zev33owWP/2Injvx8H60Yvgg583qYfCR495f7dGeGwErHzCqxtq4PfZkPe0V9dUwJ++Ahte9OqqI/DE5d7yBagsgyevgG2LvLpiL/xtEux416vLi+CZ62H3cq8+tAueuwkKc7z64Kcw6zbvxxegbCu8eKf3/QMo3QRz7vamgzd97n3ejz14r5v3DTiyx6uL82D+t7zwAChcDa991+sTYM8KWPhPUO0LjYKP4I1/hVrfd3HXclj0ANT51o2dH8Bb3w+uS58ugbf/w/t3A28+33k4+G+bvxje/+/m9dJHgvW2Rd73y2/r6/Dx/wXrLQth5Z+C9ebXYPWThJSZReyf0aNH21lb+nOzpb8I1u/+xOyDXwbrt39gtuxXwfqNB8yW/yZYv/pds+W/DdYvf615PetOsw8fDdZPX2/20WPB+i9XNK9/P87s4z94f29oMPvN0GBdW2X2iwFmOY97dVW52f+mm+X82auPlZn9JMls1V+8urzY7EddzHL/5tUHd3r12lleXbrNqze87NX7Nnj15gVeXZTr1dve8uqCFV694z2v/nSpV+9a7tXb3/HqPau8esvrXl281qs3zvPq/Zu9et2LXn1gh1fnPevVh/d49aonvfpoiVev+KNXHz/k1R/+zuxHXc2qK7x6+W/M/qe3WV2NVy/7tdkjWcF/2w9+afbosCb1L8z+eHGwXvoLs79e07x+7pbm9Uv3NKl/bjb//zV//PV/a/55ix9uXi/5WbBe9uvm35Xlvw0uS//85T4drD96zPs38/v4D2Yb5wfrFX8y27ooWK98wiz/3WC96i9mO5cF69VPmRXkBOu8Z8yK8ry/NzaarX3B+06YmTXUm62fY7Z/i1fX13qf7V92ddVmW94wO7TLq2urzLa9bXak0KtrKr3vTfleX33M66WiNFgXrDCrPOjV1RVmhavNjh8O1sVrvO+8v973SXDZVx/1vlc1lcG6bLvXh//5Bz8NfjeqK8wO7fbmw//5R4rM6uuC9dF93nz7+68o9dZJM7Pa416vjY3B+a06Eqzrqr0e/OpqvPf0q68N9mbmfa6/F/+/t/+zzbzP9b93CwJy7SS/q2H/Yf8yf75UKMz9utncbwTrOX/v/dD7vXRP8xX9xRleUATq6Wbv/Xfzx5uGyoszzD76v+bv5//R9n/emueb9+P/kW5sNJv3LbPNC726vtZswf3ej6+Z96V688Hgil5zzGzxD4MrevVRsyWPeCuTmfel/fBRs5JNXn38kBcwZfleXXnQ+2E4tDtYr5/jhYu/3rwguCJXHvR68a/IlQfNdn0YXHGPH/KCxb8yVB3xPjuwoh71giqwYlY2XxFrq7z38K+I9XXeyhaClUMkFn1RKDiL4NtxZmdnm8Y+EhE5M865PDPLPtFjOqYgIiIBCgUREQlQKIiISIBCQUREAhQKIiISEJGhoAHxRERCIyJDwTQgnohISERkKIiISGgoFEREJEChICIiAQoFEREJUCiIiEiAQkFERAIUCiIiEqBQEBGRAIWCiIgEKBRERCRAoSAiIgEKBRERCVAoiIhIgEJBREQCFAoiIhKgUBARkQCFgoiIBLSqUHDO3eic+4tz7jXn3NXh7kdEJNaEPBScc08550qdcxs/M32Sc26bc26Hc+5BADN71cy+Cfw9cHuoexMRkebOxZbC08CkphOcc3HAH4DJwGDgTufc4CZP+U/f4yIicg6FPBTMbBlw6DOTxwI7zGynmdUCLwJTnefnwJtmtuZE7+ecm+mcy3XO5ZaVlYW2eRGRGBOuYwppQGGTusg37R+BK4FbnXPfPtELzewJM8s2s+zk5OTQdyoiEkPahulz3QmmmZk9Bjx2rpsRERFPuLYUioCMJnU6sDdMvYiIiE+4QmE1cL5zrp9zrh1wB7DgdF/snLveOfdEeXl5yBoUEYlF5+KU1NnACmCgc67IOfd1M6sH/gF4G9gCzDGzTaf7nma20Mxmdu3aNTRNi4jEqJAfUzCzO08yfRGwKNSfLyIip69VXdEsIiLhFZGhoGMKIiKhEZGhoGMKIiKhEZGhICIioaFQEBGRgIgMBR1TEBEJjYgMBR1TEBEJjYgMBRERCQ2FgoiIBCgUREQkICJDQQeaRURCIyJDQQeaRURCIyJDQUREQkOhICIiAQoFEREJUCiIiEhARIaCzj4SEQmNiAwFnX0kIhIaERkKIiISGgoFEREJUCiIiEiAQkFERAIUCiIiEhCRoaBTUkVEQiMiQ0GnpIqIhEZEhoKIiISGQkFERAIUCiIiEqBQEBGRAIWCiIgEKBRERCRAoSAiIgERGQq6eE1EJDQiMhR08ZqISGhEZCiIiEhoKBRERCRAoSAiIgEKBRERCVAoiIhIgEJBREQCFAoiIhKgUBARkQCFgoiIBCgUREQkICZD4aMdB/jfRVvYc/B4uFsREWlVIjIUvuyAeJ8Ul/PXD3dx+a+W8LWnV7N0WymNjdbCXYqIRB5nFrk/htnZ2Zabm3tWry0pr2bWqj3MWrmHA8dq6JfUibvH9+WW0el07RDfwp2KiLQezrk8M8s+4WOxGgp+tfWNvLlxH8+uKCCv4DAd4uO4aVQa90zoy6DeXVqoUxGR1kOhcJo2FpfzzMe7WbB+LzX1jYzr14N7v5LFVYNTiI+LyD1tIiKfo1A4Q4cra5mTW8hzOQUUHa6id5cEpo/L5I6xmSQntm/xzxMROZcUCmepodFYsrWUZ1bsZnn+AeLjHNdd1Ie7J2QxKrMbzrmQfbaISKh8USi0PdfNRJK4No4rB6dw5eAUPi07xnMrCnglr4hX1+1laFoX7pmQxQ3DU0mIjwt3qyIiLUJbCmeosqaeeWuLeW7FbrbvP0b3jvFMG5PBjHF9yejR8Zz2IiJyNrT7KATMjJydh3jm4928s2U/jWZcMagX90zI4pIBSbRpo11LItI6afdRCDjnmHBeTyac15O9R6qYtXIPL67ew7tbVtEvqRMzxvflVl3zICIR5rS2FJxz9wN/AyqAJ4GRwINmtji07X2xcG4pnEhNfQNvflLCsyt2s2bPEV3zICKt0pfefeScW29mw51z1wDfBR4G/mZmo1q21TPT2kKhqY3F5Ty7YjevrfOueRjbrwf3TOjLNUN665oHEQmrlgiFDWY2zDn3O2Cpmc13zq01s5Et3eyZaM2h4Oe/5uH5lQUUHqqiV2J77hqXyV1jM+nVJSHc7YlIDGqJUPgbkAb0A4YDcXjhMLolGz1TkRAKfg2NxgfbS3nm4wI+2F5G2zaOSUN7c+9Xssju213XPIjIOdMSodAGGAHsNLMjzrkeQLqZbWjZVs9MJIVCU7sPVPJ8TgFzcgs5Wl3PoN6J3DMhixtHptKxnY79i0hotUQoXAysM7NK59wMYBTwOzMraNlWz0ykhoLf8dp6FqzbyzMrCtiy7yiJCW25dXQ6d4/vS//kzuFuT0SiVIscU8DbbTQMeA74K3CzmV3eko2eqUgPBT8zI6/gMM+uKODNjfuoazAuPT+JeyZk8dVBvYjTNQ8i0oJa4jqFejMz59xUvC2Evzrn7m25FmObc47srB5kZ/WgtOJCXlxVyKyVe/jms7mkdevA9PGZ3J6dQc/OGoxPRELrdLcUPgDeAr4GXAqU4e1Ouii07X2xaNlSOJH6hkbe2byfZ1cUsGLnQdrFtWHKsD7cPaEvIzI0GJ+InL2W2H3UG7gLWG1my51zmcBEM3u2BZvsD/wA6Gpmt57Oa6I5FJravr+C51YUMG9NEZW1DVyU1pW7J/TVYHwiclZaZOwj51wKMMZXrjKz0tN4zVPAFKDUzIY2mT4J+B3eqa1PmtkjTR6bq1A4sYrqOuavLea5FQXklx6jW8d4pmV7g/Fl9tRgfCJyelpiS2Ea8EtgKeDwdiF9z8zmnuJ1lwHHgGf9oeCciwO2A1cBRcBq4E4z2+x7XKFwCv7B+J7L2c3bm7zB+CZekMw9E7K4/IJkDcYnIl+oJQ40/wAY4986cM4lA+8CXxgKZrbMOZf1mcljgR1mttP3Xi8CU4HNp9lLzGs6GF9JeTWzVu1h9qo93Pf0ajJ6dGDGuL5My86ge6d24W5VRCLM6Q7C0+Yzu4sOnsFrPysNKGxSFwFpzrmezrnHgZHOuYdO9mLn3EznXK5zLresrOwsW4gevbsm8K9XXcBH3/8qv79rJH26duBnb25l/M/e44GX17O+8Ei4WxSRCHK6WwpvOefeBmb76tuBRWf5mSfat2FmdhD49qlebGZPAE+At/voLHuIOu3atmHKsFSmDEtla8lRns8pYN6aYubmFTE8vSt3T8hiyrA+OjAtIl/oTA403wJcjPejvszM5p/m67KA15scU5gA/JeZXeOrHwIws5+dafOxekzhdFVU1zFvTTHP5RSwQwemRcQnrHdeO0EotMU70HwFUIx3oPkuM9t0Bu95PXD9gAEDvpmfn9/iPUcbM2PFzoM8t6KAxZu9A9OXX5DM3eP7MnGgrpgWiTVnHQrOuQrgRE9weLt8vvDOMc652cBEIAnYD/zIdzX0tcCjeKekPmVmPz2dGfksbSmcuZLyamb7DkyXVtSQ3r0D08f1ZVp2uq6YFokRukezfE5dQyOLN+3nuZzd5Ow8FLhiesaEvozUFdMiUU2hIF9o+/4KXsgp4JU1xRyrqWdIahfuHt+XG0ZoKG+RaBR1oaBjCqFxrKaeV9cW83xOAVtLKgJDec8Y35fzNJS3SNSIulDw05ZCaJgZuQWHea7JUN4XD+jJ3eP7cuWFKbTVPaZFIppCQc5aWUUNc3ILeSGngL3l1aR0ac+dYzO5c2wmKbrHtEhEUijIl9bQaLy/tZTnc7x7TMe1cVw9OIW7x/dlwnk9dWBaJIJEXSjomEJ47T5QyaxVe5iTW8iR43Wcl9yJ6eP6csvodLp2iA93eyJyClEXCn7aUgiv6roGXt+wj+dzClhXeISE+DZMHZ7GjPF9uSi9a7jbE5GTUChIyG0sLuf5nAJeW7eXqroGhmd0Y8a4TK7XjYBEWh2Fgpwz5VV1zF9TxHM5BXxaVknXDvHcOjqd6eMy6a/TWkVaBYWCnHP+GwE9v7KAtzeWUN/ondY6Y1xfrhycQrxOaxUJm6gLBR1ojiylFdXMWV3I7FWFFB+pIqVLe24fk8mdYzPo07VDuNsTiTlRFwp+2lKILA2NxpKtpTyXU8Cy/DLaOMcVg3oxY3xfLhmQpNuIipwjLXE7TpEvLa6N48rBKVw5OIU9B48HTmtdvHk/fXt25K6xmdyWnUEP3UZUJGy0pSBhVVPfwFsbS3g+p4DVuw/Trm0brruoD9PHZTK6b3ddFCcSAtp9JBFhW0kFL6z0biN6rKaeQb0TmT4ukxtHppGYoIviRFqKQkEiSmVNPQvW7+X5nAI27T1Kx3ZxTB2RxvRxmQxN00VxIl9W1IWCzj6KDWbGhiLvoriFG/ZSXdfI8IxuTB+XyfXDUunQThfFiZyNqAsFP20pxI7y43XMW1vECyv3sKP0GIkJbblllHdR3PkpieFuTySiKBQkapgZq3YdYtaqPbz5SQm1DY2MzerB9PGZTBram/ZttfUgcioKBYlKB4/VMDeviFmr9lBw8DjdO8ZzW3YGd47NpF9Sp3C3J9JqKRQkqjU2Gh99eoBZK/ewePN+GnxDatw1ti9XDU6hXVsNqSHSlEJBYkbp0Wrm5AaH1Ejq3J5p2encOTaTjB4dw92eSKugUJCY09BoLNtexgsrC3h/aykGXHp+MneNzeSKC3tpQD6JaVEXCjolVc7E3iNVvLS6kJdWF1JytJpeie25fUwGt4/JIL27th4k9kRdKPhpS0HORH1DI0u2lTF71R6WbCsFYOIFydw5NpOvDupFW209SIxQKIh8RvGRKl5atYeXcgvZf7SGlC7tmZatrQeJDQoFkZPwbz3MWlnA0u1lAFzu23q4QlsPEqUUCiKnoejwceasLgxsPfiPPUzLztCZSxJVFAoiZ+BEWw/emUsZXHGhbiUqkU+hIHKWio9UeVsPvjOXkjq357bsdO4Yk0HfnrpqWiKTQkHkS6pvaOSD7WXMXlXI+1v302hw8YCe3DEmk6uHpGjMJYkoCgWRFlRSXs3LuYW8uNq7arpHp3bcPDKNO8ZmMqBX53C3J3JKURcKunhNWoPGRmP5jgO8uGoP72zeT32jMSarO3eMyeTai/rofg/SakVdKPhpS0FaiwPHanglr4gXVxey60AliQltuXFEGrePydDd4qTVUSiInCNmxspdh3hpdSGLPtlHTX0jF6V15fYxGdwwIpUuute0tAIKBZEwKD9ex6vripm9ag9bSypIiG/DdRelcsfYDLL7dsc5F+4WJUYpFETCyMxYX1TOS6sLWbh+L8dq6umf3InbszO4eVQ6yYntw92ixBiFgkgrcby2ntc37GPO6kJyCw7Tto3jigt7cfuYDC47P1nDasg5oVAQaYV2lFbw0upC5q0p5mBlLb27JHDr6HSmZWeQ2VPDakjoKBREWrHa+kbe37qfl1YX8sH2MhoNxvfvwe1jMpg8tA8J8Tq1VVqWQkEkQuwrr+KVvCJeyi2k8FAViQltmToilWnZGVyU1lUHp6VFKBREIkxjo5Gz8yAv5xUFTm0d1DuRadkZ3DgyjR6d2oW7RYlgCgWRCFZeVcfC9Xt5ObeQ9UXltItrw5WDe3FbtndwOq6Nth7kzCgURKLE1pKjzFldxKvrijlUWUtKl/bcMiqdW0en0z9Z4y7J6VEoiEQZ/8HpOblFLN1WSqNBdt/u3JadznXDUuncvm24W5RWLOpCQQPiiQSVHq1m3tpi5uQWsrOskg7xcUy+qDe3jc5gXL8etNHuJfmMqAsFP20piASZGWv2HGFuXhGvr99LRU096d07BHYv6Zai4qdQEIkxVbUNvL2phLl5RXz06QHMd+3DraMzmDy0N520eymmKRREYljR4ePMX1PM3DVFFBw8Tsd2cVx7UR9uGZWu3UsxSqEgIpgZuQWHeSWviNc37OOYb/fSzSPTuHlUOllJuud0rFAoiEgz/t1Lr6wp4sMd3u6l7L7duWV0OtcN66P7PkQ5hYKInNS+8irmry3mlbwiPi2rpH3bNlw1OIVbRqVz6flJGrk1CikUROSUzIwNReW8sqaIhev3cvh4HUmd2zN1RCo3j0pjcJ8uGnspSigUROSM1NY3snRbKa+sKeL9raXUNRgDUxK5aVQaN45Io3fXhHC3KF+CQkFEztrhylpe/2Qf89YUsXbPEZyDr5zXk5tGpjNpaG9dPR2BFAoi0iJ2Hahk/tpiXl1bzJ5Dx+kQH8fVQ1K4aWQalwzQ8YdIoVAQkRZlZuQVHGb+2mJe37CP8qo6kjq34/rhqdw4Io1h6br3Q2umUBCRkKmpb2DJ1jJeW1fMe1tKqW1opH9SJ6aOSOPGkan07anrH1qbmAqFuro6ioqKqK6uDlNX50ZCQgLp6enEx+t8cmk9yo/X8ebGfby6rpicnYcAGJnZjanDU5kyPJWkzu3D3KFAjIXCrl27SExMpGfPnlG7+WpmHDx4kIqKCvr16xfudkROaO+RKhas38ura4vZWlJBXBvHJQOSmDoilauH6AB1OMVUKGzZsoVBgwZFbSD4mRlbt27lwgsvDHcrIqe0raSCV9cVs2DdXoqPVJEQ34YrLkzhhuGpTByYTPu2ceFuMaZ8UShEZVRHeyBAbMyjRI+BvRP5/qRBfO/qgazZc5hX1xWz6JMS3tiwj8SEtkwe2psbhqcx4byeur1omEVlKIhI69SmjSM7qwfZWT340fVD+GjHARas28sbG/YxJ7eIpM7tmTKsD9cPT2VUZjf9z08YKBRa2JEjR5g1axbf+c53zuh11157LbNmzaJbt24h6kykdYmPa8PEgb2YOLAX1XUNvL+1lAXr9jJr1R6e/ng3ad06MGV4H64flsqQVA2xca60mmMKzrlOwB+BWmCpmb1wqtec7JhCOPez7969mylTprBx48Zm0xsaGoiLa9n9puGeV5FQqKiu453N+1mwfi8f5h+gvtHol9QpsAVxQUpiuFuMeGE7puCcewqYApSa2dAm0ycBvwPigCfN7BHgZmCumS10zr0EnDIUTuXHCzexee/RL/s2zQxO7cKPrh9y0scffPBBPv30U0aMGEF8fDydO3emT58+rFu3js2bN3PjjTdSWFhIdXU1999/PzNnzgQgKyuL3Nxcjh07xuTJk7nkkkv4+OOPSUtL47XXXqNDhw4tOh8irVViQjw3j0rn5lHpHK6s5e1NJSxYv5c/LNnB/72/gwtSOnPdRalMGd6H85I7h7vdqBPq3UdPA78HnvVPcM7FAX8ArgKKgNXOuQVAOvCJ72kNIe4rZB555BE2btzIunXrWLp0Kddddx0bN24MnDr61FNP0aNHD6qqqhgzZgy33HILPXv2bPYe+fn5zJ49m7/85S9MmzaNV155hRkzZoRjdkTCqnundtwxNpM7xmZSVlHDWxv3sXDDPh59bzu/fXc7g3onMmVYH669qA/9FRAtIqShYGbLnHNZn5k8FthhZjsBnHMvAlPxAiIdWAe0yAAqX/R/9OfK2LFjm11L8NhjjzF//nwACgsLyc/P/1wo9OvXjxEjRgAwevRodu/efc76FWmtkhPbc/eELO6ekEVJeTVvbtzH6xv28avF2/nV4u1c2KdLICD66S5yZy0cB5rTgMImdREwDngM+L1z7jpg4cle7JybCcwEyMzMDGGbLaNTp+CXc+nSpbz77rusWLGCjh07MnHixBNeed2+ffCqz7i4OKqqqs5JryKRonfXBO67uB/3XdyPvUeqeHNjCW9s2Msv397GL9/exoV9unDt0N5cO0y7mM5UOELhRKcQmJlVAved6sVm9gTwBHgHmlu4ty8tMTGRioqKEz5WXl5O9+7d6dixI1u3biUnJ+ccdycSfVK7deDrl/Tj65d4AfHWxhIWfbKPX7+znV+/s52BKYlMGtqbay/qwwUpnXUW0ymEIxSKgIwmdTqwNwx9hETPnj25+OKLGTp0KB06dCAlJSXw2KRJk3j88ccZNmwYAwcOZPz48WHsVCT6pHbrwNcu6cfXLulHSXk1b23cx6KNJTz2fj6/ey+f/kmdmDS0N5OH9mFomk5zPZGQn5LqO6bwuv/sI+dcW2A7cAVQDKwG7jKzTWfwntcD1w8YMOCb+fn5zR6LpdM0Y2leRb6M0opq3t60n7c27iNn5yEaGo307h2YNKQ3k4b2ZlRmd9rE0JXUYRv7yDk3G5gIJAH7gR+Z2V+dc9cCj+KdkvqUmf30bN6/NV6ncC7F0ryKtJRDlbW8u3k/b27cx4c7DlDXYCQntufqwSlcM6Q34/v3pF3b6L5ZUNiuUzCzO08yfRGwKJSfLSJyIj06tWPamAymjcngaHUdS7aW8vamEuavLeaFlXtITGjLFYN6cc2Q3lx2QTKdYmw019iaWxGRJrokxDN1RBpTR6RRXdfA8vwDvL2phHe37OfVdXtp17YNlw5I4uohKVxxYUpM3A8iIkOhyTGFcLciIlEiIT6OqwancNXgFOobGlm9+zCLN5eweNN+3ttainOfMCqze+A50Xqqa6sZ++hs6JhC7MyrSLiYGZv3HeWdzft5Z/N+NvmGzumf3ImrBqdw5YUpjMrsHlFDfsfc/RRERFqKc44hqV0ZktqVf77yAoqPVPGuLyCe+nAXf/5gJz06tWPiwGSuvDCFS89PIjEhcm+Tq1BoYWc7dDbAo48+ysyZM+nYsWMIOhORlpDWrQP3fiWLe7+SRUV1Hcu2H+CdzSW8t6WUeWuKiY9zjO/fk68O6sUVg1LI7BlZ63NE7j5qzdcpnGzo7NPhHyk1KSnptJ4f7nkVkaD6hkbyCg7z3tZS3tuyn0/LKgEY0KszXx3Ui78b2IvsrO7Ex4X/dNeYu0dzsx/Kv10HI+6CkdOhoQ6evRFG3QPDb4fa4/DCbTDmazD0Fqguh9l3wbhvweAboPIgzLkHvvIPMHAyVBg7WE0AAAltSURBVOyHxBS+yB133MFrr73GwIEDueqqq+jVqxdz5syhpqaGm266iR//+MdUVlYybdo0ioqKaGho4OGHH2b//v088MADDBw4kKSkJJYsWXLK+VcoiLReuw9U8t7WUpZsLWXlroPUNRiJCW257PxkJg5M5vKByfRKTAhLbzqmcA41HTp78eLFzJ07l1WrVmFm3HDDDSxbtoyysjJSU1N54403AG9MpK5du/Kb3/yGJUuWnPaWgoi0XllJnQJjMh2rqefD/AMs3VbKkm2lvPHJPgCGpnVh4gW9+LtByQxP70bbVrAVEf2hcN8bwb/HxTev23VsXid0bV536tm8PsVWwmctXryYxYsXM3LkSACOHTtGfn4+l156KQ888ADf//73mTJlCpdeeukZva+IRJbO7dsyaag3pIb/bKal28pYuq2UPy7dwe+X7KBrh3guOT+JiRckc9kFyaR0Cc9WRPSHQhiZGQ899BDf+ta3PvdYXl4eixYt4qGHHuLqq6/mhz/8YRg6FJFzrenZTN/9uwGUH69j+Y4yPthWxgfby3hjg7cVMah3IpddkMxl5yeTndWdhPiWvZ3vyURkKLTmi9eaDp19zTXX8PDDDzN9+nQ6d+5McXEx8fHx1NfX06NHD2bMmEHnzp15+umnm71Wu49EYkfXjvFMGZbKlGGpmBlb9lWwLL+MZdvL+NtHu3hi2U4S4tswvn9PLj0/mcvOT2JAr9ANAR6RoWBmC4GF2dnZ3wx3L5/VdOjsyZMnc9dddzFhwgQAOnfuzPPPP8+OHTv43ve+R5s2bYiPj+dPf/oTADNnzmTy5Mn06dPntA40i0h0cc4xOLULg1O78O3Lz6Oypp6cnQdZnn+AZdvL+O9tmwFI6dKen0wdyjVDerd8D1F/9lEUi6V5FREoOnycD/MPsHzHAWZe2p/hGd3O6n109pGISBRI796RO8ZmcsfY0N2KOPznP4mISKsRlaEQybvETlcszKOInHsRGQrOueudc0+Ul5d/7rGEhAQOHjwY1T+aZsbBgwdJSAjPecwiEr2i7kBzXV0dRUVFVFdXh6mrcyMhIYH09HTi4yN3NEYRCY+YOtAcHx9Pv379wt2GiEhEisjdRyIiEhoKBRERCVAoiIhIQEQfaHbOlQEFZ/CSJOBAiNppzWJxvmNxniE25zsW5xm+3Hz3NbPkEz0Q0aFwppxzuSc74h7NYnG+Y3GeITbnOxbnGUI339p9JCIiAQoFEREJiLVQeCLcDYRJLM53LM4zxOZ8x+I8Q4jmO6aOKYiIyBeLtS0FERH5AgoFEREJiJlQcM5Ncs5tc87tcM49GO5+QsE5l+GcW+Kc2+Kc2+Scu983vYdz7h3nXL7vv93D3WtLc87FOefWOude99X9nHMrffP8knOuXbh7bGnOuW7OubnOua2+ZT4h2pe1c+5ffN/tjc652c65hGhc1s65p5xzpc65jU2mnXDZOs9jvt+2Dc65UV/ms2MiFJxzccAfgMnAYOBO59zg8HYVEvXAv5nZhcB44Lu++XwQeM/Mzgfe89XR5n5gS5P658BvffN8GPh6WLoKrd8Bb5nZIGA43vxH7bJ2zqUB/wRkm9lQIA64g+hc1k8Dkz4z7WTLdjJwvu/PTOBPX+aDYyIUgLHADjPbaWa1wIvA1DD31OLMbJ+ZrfH9vQLvRyINb16f8T3tGeDG8HQYGs65dOA64Elf7YCvAnN9T4nGee4CXAb8FcDMas3sCFG+rPFGdu7gnGsLdAT2EYXL2syWAYc+M/lky3Yq8Kx5coBuzrk+Z/vZsRIKaUBhk7rINy1qOeeygJHASiDFzPaBFxxAr/B1FhKPAv8ONPrqnsARM6v31dG4vPsDZcDffLvNnnTOdSKKl7WZFQO/AvbghUE5kEf0L2u/ky3bFv19i5VQcCeYFrXn4jrnOgOvAP9sZkfD3U8oOeemAKVmltd08gmeGm3Luy0wCviTmY0EKomiXUUn4tuHPhXoB6QCnfB2nXxWtC3rU2nR73ushEIRkNGkTgf2hqmXkHLOxeMFwgtmNs83eb9/c9L339Jw9RcCFwM3OOd24+0W/CrelkM33y4GiM7lXQQUmdlKXz0XLySieVlfCewyszIzqwPmAV8h+pe138mWbYv+vsVKKKwGzvedpdAO7+DUgjD31OJ8+9L/Cmwxs980eWgBcK/v7/cCr53r3kLFzB4ys3Qzy8Jbru+b2XRgCXCr72lRNc8AZlYCFDrnBvomXQFsJoqXNd5uo/HOuY6+77p/nqN6WTdxsmW7ALjHdxbSeKDcv5vpbMTMFc3OuWvx/g8yDnjKzH4a5pZanHPuEmA58AnB/ev/gXdcYQ6Qibdi3WZmnz2IFfGccxOBB8xsinOuP96WQw9gLTDDzGrC2V9Lc86NwDu43g7YCdyH9z96UbusnXM/Bm7HO9NuLfANvP3nUbWsnXOzgYl4w2PvB34EvMoJlq0vIH+Pd7bSceA+M8s90fue1mfHSiiIiMipxcruIxEROQ0KBRERCVAoiIhIgEJBREQCFAoiIhKgUBA5h5xzE/0juYq0RgoFEREJUCiInIBzboZzbpVzbp1z7s+++zUcc8792jm3xjn3nnMu2ffcEc65HN9Y9vObjHM/wDn3rnNuve815/nevnOT+yC84Lv4COfcI865zb73+VWYZl1inEJB5DOccxfiXTV7sZmNABqA6XgDsK0xs1HAB3hXmQI8C3zfzIbhXU3un/4C8AczG443Ro9/6IGRwD/j3dujP3Cxc64HcBMwxPc+/xPauRQ5MYWCyOddAYwGVjvn1vnq/nhDh7zke87zwCXOua5ANzP7wDf9GeAy51wikGZm8wHMrNrMjvues8rMisysEVgHZAFHgWrgSefczXjDFYiccwoFkc9zwDNmNsL3Z6CZ/dcJnvdFY8ScaDhjv6bj8jQAbX33AxiLN8LtjcBbZ9izSItQKIh83nvArc65XhC4N25fvPXFPxrnXcCHZlYOHHbOXeqbfjfwge8+FkXOuRt979HeOdfxZB/ouwdGVzNbhLdraUQoZkzkVNqe+ikiscXMNjvn/hNY7JxrA9QB38W7kc0Q51we3l2/bve95F7gcd+Pvn+0UvAC4s/OuZ/43uO2L/jYROA151wC3lbGv7TwbImcFo2SKnKanHPHzKxzuPsQCSXtPhIRkQBtKYiISIC2FEREJEChICIiAQoFEREJUCiIiEiAQkFERAL+P3RAawHxfCjoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_and_plot(poly_features[0:2, :], poly_features[n_train:, :], labels[0:2], labels[n_train:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_4tfoquk",
    "id": "A1A8C28C0A204B04AD8B5FFF9C2D2F5B",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# 权重衰减\n",
    "## 方法  \n",
    "权重衰减等价于 $L_2$ 范数正则化（regularization）。正则化通过为模型损失函数添加惩罚项使学出的模型参数值较小，是应对过拟合的常用手段。\n",
    "\n",
    "##  L2 范数正则化（regularization）\n",
    "$L_2$范数正则化在模型原损失函数基础上添加$L_2$范数惩罚项，从而得到训练所需要最小化的函数。$L_2$范数惩罚项指的是模型权重参数每个元素的平方和与一个正的常数的乘积。以线性回归中的线性回归损失函数为例\n",
    "\n",
    "\n",
    "$$\n",
    " \\ell(w_1, w_2, b) = \\frac{1}{n} \\sum_{i=1}^n \\frac{1}{2}\\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right)^2 \n",
    "$$\n",
    "\n",
    "\n",
    "其中$w_1, w_2$是权重参数，$b$是偏差参数，样本$i$的输入为$x_1^{(i)}, x_2^{(i)}$，标签为$y^{(i)}$，样本数为$n$。将权重参数用向量$\\boldsymbol{w} = [w_1, w_2]$表示，带有$L_2$范数惩罚项的新损失函数为\n",
    "\n",
    "\n",
    "$$\n",
    "\\ell(w_1, w_2, b) + \\frac{\\lambda}{2n} |\\boldsymbol{w}|^2,\n",
    "$$\n",
    "\n",
    "\n",
    "其中超参数$\\lambda > 0$。当权重参数均为0时，惩罚项最小。当$\\lambda$较大时，惩罚项在损失函数中的比重较大，这通常会使学到的权重参数的元素较接近0。当$\\lambda$设为0时，惩罚项完全不起作用。上式中$L_2$范数平方$|\\boldsymbol{w}|^2$展开后得到$w_1^2 + w_2^2$。\n",
    "有了$L_2$范数惩罚项后，在小批量随机梯度下降中，我们将线性回归一节中权重$w_1$和$w_2$的迭代方式更改为\n",
    "\n",
    "\n",
    "$$\n",
    " \\begin{aligned} w_1 &\\leftarrow \\left(1- \\frac{\\eta\\lambda}{|\\mathcal{B}|} \\right)w_1 - \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}}x_1^{(i)} \\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right),\\\\ w_2 &\\leftarrow \\left(1- \\frac{\\eta\\lambda}{|\\mathcal{B}|} \\right)w_2 - \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}}x_2^{(i)} \\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right). \\end{aligned} \n",
    "$$\n",
    "\n",
    "\n",
    "可见，$L_2$范数正则化令权重$w_1$和$w_2$先自乘小于1的数，再减去不含惩罚项的梯度。因此，$L_2$范数正则化又叫权重衰减。权重衰减通过惩罚绝对值较大的模型参数为需要学习的模型增加了限制，这可能对过拟合有效。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_6r7el1m",
    "id": "86B363E2493A4A2980C5E00729B925F5",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 高维线性回归实验从零开始的实现\n",
    "下面，我们以高维线性回归为例来引入一个过拟合问题，并使用权重衰减来应对过拟合。设数据样本特征的维度为$p$。对于训练数据集和测试数据集中特征为$x_1, x_2, \\ldots, x_p$的任一样本，我们使用如下的线性函数来生成该样本的标签：\n",
    "\n",
    "\n",
    "$$\n",
    " y = 0.05 + \\sum_{i = 1}^p 0.01x_i + \\epsilon \n",
    "$$\n",
    "\n",
    "\n",
    "其中噪声项$\\epsilon$服从均值为0、标准差为0.01的正态分布。为了较容易地观察过拟合，我们考虑高维线性回归问题，如设维度$p=200$；同时，我们特意把训练数据集的样本数设低，如20。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "graffitiCellId": "id_m6mthxc",
    "id": "51E803AFA42047D48605A9042CD266F8",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.2.0\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import numpy as np\n",
    "import sys\n",
    "sys.path.append(\"/home/kesci/input\")\n",
    "import d2lzh1981 as d2l\n",
    "\n",
    "print(torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_d4sywej",
    "id": "5AC8DFCEDD734B8289B2CE7D06C82B6F",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 初始化模型参数\n",
    "与前面观察过拟合和欠拟合现象的时候相似，在这里不再解释。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "graffitiCellId": "id_3ymafxv",
    "id": "D66AE4FB939640948F151F6FF7C1740E",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "n_train, n_test, num_inputs = 20, 100, 200\n",
    "true_w, true_b = torch.ones(num_inputs, 1) * 0.01, 0.05\n",
    "\n",
    "features = torch.randn((n_train + n_test, num_inputs))\n",
    "labels = torch.matmul(features, true_w) + true_b\n",
    "labels += torch.tensor(np.random.normal(0, 0.01, size=labels.size()), dtype=torch.float)\n",
    "train_features, test_features = features[:n_train, :], features[n_train:, :]\n",
    "train_labels, test_labels = labels[:n_train], labels[n_train:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "graffitiCellId": "id_xd0w3ub",
    "id": "E1707F6923A742F69F84896E84B463FE",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 定义参数初始化函数，初始化模型参数并且附上梯度\n",
    "def init_params():\n",
    "    w = torch.randn((num_inputs, 1), requires_grad=True)\n",
    "    b = torch.zeros(1, requires_grad=True)\n",
    "    return [w, b]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_ejz9flm",
    "id": "01F0DAF1CD6F48BA92BAAE9F7554EA28",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 定义L2范数惩罚项"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "graffitiCellId": "id_kkiguji",
    "id": "4709112E91504EF68681066730402A03",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def l2_penalty(w):\n",
    "    return (w**2).sum() / 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_99qaypt",
    "id": "06C980C7C2534A728058234C02210A34",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 定义训练和测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "graffitiCellId": "id_6crgl61",
    "id": "3A369C128AE64990854186DAD85DDD7F",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "batch_size, num_epochs, lr = 1, 100, 0.003\n",
    "net, loss = d2l.linreg, d2l.squared_loss\n",
    "\n",
    "dataset = torch.utils.data.TensorDataset(train_features, train_labels)\n",
    "train_iter = torch.utils.data.DataLoader(dataset, batch_size, shuffle=True)\n",
    "\n",
    "def fit_and_plot(lambd):\n",
    "    w, b = init_params()\n",
    "    train_ls, test_ls = [], []\n",
    "    for _ in range(num_epochs):\n",
    "        for X, y in train_iter:\n",
    "            # 添加了L2范数惩罚项\n",
    "            l = loss(net(X, w, b), y) + lambd * l2_penalty(w)\n",
    "            l = l.sum()\n",
    "            \n",
    "            if w.grad is not None:\n",
    "                w.grad.data.zero_()\n",
    "                b.grad.data.zero_()\n",
    "            l.backward()\n",
    "            d2l.sgd([w, b], lr, batch_size)\n",
    "        train_ls.append(loss(net(train_features, w, b), train_labels).mean().item())\n",
    "        test_ls.append(loss(net(test_features, w, b), test_labels).mean().item())\n",
    "    d2l.semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',\n",
    "                 range(1, num_epochs + 1), test_ls, ['train', 'test'])\n",
    "    print('L2 norm of w:', w.norm().item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_55gl5e9",
    "id": "BA54CB5FD4E64380AC1357E8537265DC",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 观察过拟合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "graffitiCellId": "id_6pmjf6b",
    "id": "C27406AAA0FD41C6801D55ED4B25D5EA",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L2 norm of w: 13.487003326416016\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 258.944602 180.65625\" width=\"258.944602pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 258.944602 180.65625 \r\n",
       "L 258.944602 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 55.778125 143.1 \r\n",
       "L 251.078125 143.1 \r\n",
       "L 251.078125 7.2 \r\n",
       "L 55.778125 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"md5d3f15b3d\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"62.862009\" xlink:href=\"#md5d3f15b3d\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-48\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(59.680759 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"98.729778\" xlink:href=\"#md5d3f15b3d\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 20 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-50\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(92.367278 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"134.597546\" xlink:href=\"#md5d3f15b3d\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 40 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-52\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(128.235046 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"170.465315\" xlink:href=\"#md5d3f15b3d\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 60 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 33.015625 40.375 \r\n",
       "Q 26.375 40.375 22.484375 35.828125 \r\n",
       "Q 18.609375 31.296875 18.609375 23.390625 \r\n",
       "Q 18.609375 15.53125 22.484375 10.953125 \r\n",
       "Q 26.375 6.390625 33.015625 6.390625 \r\n",
       "Q 39.65625 6.390625 43.53125 10.953125 \r\n",
       "Q 47.40625 15.53125 47.40625 23.390625 \r\n",
       "Q 47.40625 31.296875 43.53125 35.828125 \r\n",
       "Q 39.65625 40.375 33.015625 40.375 \r\n",
       "z\r\n",
       "M 52.59375 71.296875 \r\n",
       "L 52.59375 62.3125 \r\n",
       "Q 48.875 64.0625 45.09375 64.984375 \r\n",
       "Q 41.3125 65.921875 37.59375 65.921875 \r\n",
       "Q 27.828125 65.921875 22.671875 59.328125 \r\n",
       "Q 17.53125 52.734375 16.796875 39.40625 \r\n",
       "Q 19.671875 43.65625 24.015625 45.921875 \r\n",
       "Q 28.375 48.1875 33.59375 48.1875 \r\n",
       "Q 44.578125 48.1875 50.953125 41.515625 \r\n",
       "Q 57.328125 34.859375 57.328125 23.390625 \r\n",
       "Q 57.328125 12.15625 50.6875 5.359375 \r\n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \r\n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \r\n",
       "Q 6.984375 17.96875 6.984375 36.375 \r\n",
       "Q 6.984375 53.65625 15.1875 63.9375 \r\n",
       "Q 23.390625 74.21875 37.203125 74.21875 \r\n",
       "Q 40.921875 74.21875 44.703125 73.484375 \r\n",
       "Q 48.484375 72.75 52.59375 71.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-54\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(164.102815 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-54\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"206.333084\" xlink:href=\"#md5d3f15b3d\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 80 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 34.625 \r\n",
       "Q 24.75 34.625 20.71875 30.859375 \r\n",
       "Q 16.703125 27.09375 16.703125 20.515625 \r\n",
       "Q 16.703125 13.921875 20.71875 10.15625 \r\n",
       "Q 24.75 6.390625 31.78125 6.390625 \r\n",
       "Q 38.8125 6.390625 42.859375 10.171875 \r\n",
       "Q 46.921875 13.96875 46.921875 20.515625 \r\n",
       "Q 46.921875 27.09375 42.890625 30.859375 \r\n",
       "Q 38.875 34.625 31.78125 34.625 \r\n",
       "z\r\n",
       "M 21.921875 38.8125 \r\n",
       "Q 15.578125 40.375 12.03125 44.71875 \r\n",
       "Q 8.5 49.078125 8.5 55.328125 \r\n",
       "Q 8.5 64.0625 14.71875 69.140625 \r\n",
       "Q 20.953125 74.21875 31.78125 74.21875 \r\n",
       "Q 42.671875 74.21875 48.875 69.140625 \r\n",
       "Q 55.078125 64.0625 55.078125 55.328125 \r\n",
       "Q 55.078125 49.078125 51.53125 44.71875 \r\n",
       "Q 48 40.375 41.703125 38.8125 \r\n",
       "Q 48.828125 37.15625 52.796875 32.3125 \r\n",
       "Q 56.78125 27.484375 56.78125 20.515625 \r\n",
       "Q 56.78125 9.90625 50.3125 4.234375 \r\n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.734375 -1.421875 13.25 4.234375 \r\n",
       "Q 6.78125 9.90625 6.78125 20.515625 \r\n",
       "Q 6.78125 27.484375 10.78125 32.3125 \r\n",
       "Q 14.796875 37.15625 21.921875 38.8125 \r\n",
       "z\r\n",
       "M 18.3125 54.390625 \r\n",
       "Q 18.3125 48.734375 21.84375 45.5625 \r\n",
       "Q 25.390625 42.390625 31.78125 42.390625 \r\n",
       "Q 38.140625 42.390625 41.71875 45.5625 \r\n",
       "Q 45.3125 48.734375 45.3125 54.390625 \r\n",
       "Q 45.3125 60.0625 41.71875 63.234375 \r\n",
       "Q 38.140625 66.40625 31.78125 66.40625 \r\n",
       "Q 25.390625 66.40625 21.84375 63.234375 \r\n",
       "Q 18.3125 60.0625 18.3125 54.390625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-56\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(199.970584 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-56\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"242.200852\" xlink:href=\"#md5d3f15b3d\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-49\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(232.657102 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_7\">\r\n",
       "     <!-- epochs -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 56.203125 29.59375 \r\n",
       "L 56.203125 25.203125 \r\n",
       "L 14.890625 25.203125 \r\n",
       "Q 15.484375 15.921875 20.484375 11.0625 \r\n",
       "Q 25.484375 6.203125 34.421875 6.203125 \r\n",
       "Q 39.59375 6.203125 44.453125 7.46875 \r\n",
       "Q 49.3125 8.734375 54.109375 11.28125 \r\n",
       "L 54.109375 2.78125 \r\n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \r\n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \r\n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \r\n",
       "Q 5.515625 13.8125 5.515625 26.8125 \r\n",
       "Q 5.515625 40.234375 12.765625 48.109375 \r\n",
       "Q 20.015625 56 32.328125 56 \r\n",
       "Q 43.359375 56 49.78125 48.890625 \r\n",
       "Q 56.203125 41.796875 56.203125 29.59375 \r\n",
       "z\r\n",
       "M 47.21875 32.234375 \r\n",
       "Q 47.125 39.59375 43.09375 43.984375 \r\n",
       "Q 39.0625 48.390625 32.421875 48.390625 \r\n",
       "Q 24.90625 48.390625 20.390625 44.140625 \r\n",
       "Q 15.875 39.890625 15.1875 32.171875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-101\"/>\r\n",
       "      <path d=\"M 18.109375 8.203125 \r\n",
       "L 18.109375 -20.796875 \r\n",
       "L 9.078125 -20.796875 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.390625 \r\n",
       "Q 20.953125 51.265625 25.265625 53.625 \r\n",
       "Q 29.59375 56 35.59375 56 \r\n",
       "Q 45.5625 56 51.78125 48.09375 \r\n",
       "Q 58.015625 40.1875 58.015625 27.296875 \r\n",
       "Q 58.015625 14.40625 51.78125 6.484375 \r\n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \r\n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \r\n",
       "Q 20.953125 3.328125 18.109375 8.203125 \r\n",
       "z\r\n",
       "M 48.6875 27.296875 \r\n",
       "Q 48.6875 37.203125 44.609375 42.84375 \r\n",
       "Q 40.53125 48.484375 33.40625 48.484375 \r\n",
       "Q 26.265625 48.484375 22.1875 42.84375 \r\n",
       "Q 18.109375 37.203125 18.109375 27.296875 \r\n",
       "Q 18.109375 17.390625 22.1875 11.75 \r\n",
       "Q 26.265625 6.109375 33.40625 6.109375 \r\n",
       "Q 40.53125 6.109375 44.609375 11.75 \r\n",
       "Q 48.6875 17.390625 48.6875 27.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-112\"/>\r\n",
       "      <path d=\"M 30.609375 48.390625 \r\n",
       "Q 23.390625 48.390625 19.1875 42.75 \r\n",
       "Q 14.984375 37.109375 14.984375 27.296875 \r\n",
       "Q 14.984375 17.484375 19.15625 11.84375 \r\n",
       "Q 23.34375 6.203125 30.609375 6.203125 \r\n",
       "Q 37.796875 6.203125 41.984375 11.859375 \r\n",
       "Q 46.1875 17.53125 46.1875 27.296875 \r\n",
       "Q 46.1875 37.015625 41.984375 42.703125 \r\n",
       "Q 37.796875 48.390625 30.609375 48.390625 \r\n",
       "z\r\n",
       "M 30.609375 56 \r\n",
       "Q 42.328125 56 49.015625 48.375 \r\n",
       "Q 55.71875 40.765625 55.71875 27.296875 \r\n",
       "Q 55.71875 13.875 49.015625 6.21875 \r\n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \r\n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \r\n",
       "Q 5.515625 13.875 5.515625 27.296875 \r\n",
       "Q 5.515625 40.765625 12.171875 48.375 \r\n",
       "Q 18.84375 56 30.609375 56 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-111\"/>\r\n",
       "      <path d=\"M 48.78125 52.59375 \r\n",
       "L 48.78125 44.1875 \r\n",
       "Q 44.96875 46.296875 41.140625 47.34375 \r\n",
       "Q 37.3125 48.390625 33.40625 48.390625 \r\n",
       "Q 24.65625 48.390625 19.8125 42.84375 \r\n",
       "Q 14.984375 37.3125 14.984375 27.296875 \r\n",
       "Q 14.984375 17.28125 19.8125 11.734375 \r\n",
       "Q 24.65625 6.203125 33.40625 6.203125 \r\n",
       "Q 37.3125 6.203125 41.140625 7.25 \r\n",
       "Q 44.96875 8.296875 48.78125 10.40625 \r\n",
       "L 48.78125 2.09375 \r\n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \r\n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \r\n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \r\n",
       "Q 5.515625 14.109375 5.515625 27.296875 \r\n",
       "Q 5.515625 40.671875 12.859375 48.328125 \r\n",
       "Q 20.21875 56 33.015625 56 \r\n",
       "Q 37.15625 56 41.109375 55.140625 \r\n",
       "Q 45.0625 54.296875 48.78125 52.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-99\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 75.984375 \r\n",
       "L 18.109375 75.984375 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-104\"/>\r\n",
       "      <path d=\"M 44.28125 53.078125 \r\n",
       "L 44.28125 44.578125 \r\n",
       "Q 40.484375 46.53125 36.375 47.5 \r\n",
       "Q 32.28125 48.484375 27.875 48.484375 \r\n",
       "Q 21.1875 48.484375 17.84375 46.4375 \r\n",
       "Q 14.5 44.390625 14.5 40.28125 \r\n",
       "Q 14.5 37.15625 16.890625 35.375 \r\n",
       "Q 19.28125 33.59375 26.515625 31.984375 \r\n",
       "L 29.59375 31.296875 \r\n",
       "Q 39.15625 29.25 43.1875 25.515625 \r\n",
       "Q 47.21875 21.78125 47.21875 15.09375 \r\n",
       "Q 47.21875 7.46875 41.1875 3.015625 \r\n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \r\n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \r\n",
       "Q 10.6875 0.296875 5.421875 2 \r\n",
       "L 5.421875 11.28125 \r\n",
       "Q 10.40625 8.6875 15.234375 7.390625 \r\n",
       "Q 20.0625 6.109375 24.8125 6.109375 \r\n",
       "Q 31.15625 6.109375 34.5625 8.28125 \r\n",
       "Q 37.984375 10.453125 37.984375 14.40625 \r\n",
       "Q 37.984375 18.0625 35.515625 20.015625 \r\n",
       "Q 33.0625 21.96875 24.703125 23.78125 \r\n",
       "L 21.578125 24.515625 \r\n",
       "Q 13.234375 26.265625 9.515625 29.90625 \r\n",
       "Q 5.8125 33.546875 5.8125 39.890625 \r\n",
       "Q 5.8125 47.609375 11.28125 51.796875 \r\n",
       "Q 16.75 56 26.8125 56 \r\n",
       "Q 31.78125 56 36.171875 55.265625 \r\n",
       "Q 40.578125 54.546875 44.28125 53.078125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-115\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(135.595313 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\r\n",
       "      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"mc846740ebd\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#mc846740ebd\" y=\"122.626361\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- $\\mathdefault{10^{-10}}$ -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-8722\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 126.42558)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(231.391602 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#mc846740ebd\" y=\"95.26694\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- $\\mathdefault{10^{-7}}$ -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 8.203125 72.90625 \r\n",
       "L 55.078125 72.90625 \r\n",
       "L 55.078125 68.703125 \r\n",
       "L 28.609375 0 \r\n",
       "L 18.3125 0 \r\n",
       "L 43.21875 64.59375 \r\n",
       "L 8.203125 64.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-55\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(25.278125 99.066158)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-55\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#mc846740ebd\" y=\"67.907518\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- $\\mathdefault{10^{-4}}$ -->\r\n",
       "      <g transform=\"translate(25.278125 71.706737)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-52\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#mc846740ebd\" y=\"40.548097\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- $\\mathdefault{10^{-1}}$ -->\r\n",
       "      <g transform=\"translate(25.278125 44.347316)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#mc846740ebd\" y=\"13.188676\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- $\\mathdefault{10^{2}}$ -->\r\n",
       "      <g transform=\"translate(31.178125 16.987894)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_13\">\r\n",
       "     <!-- loss -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-108\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798438 84.807812)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-108\"/>\r\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_12\">\r\n",
       "    <path clip-path=\"url(#p51a6f5e4d6)\" d=\"M 64.655398 19.681609 \r\n",
       "L 66.448786 25.35468 \r\n",
       "L 68.242175 30.09269 \r\n",
       "L 70.035563 34.094217 \r\n",
       "L 71.828951 37.872524 \r\n",
       "L 73.62234 41.688539 \r\n",
       "L 75.415728 45.400736 \r\n",
       "L 77.209117 49.171548 \r\n",
       "L 79.002505 52.894871 \r\n",
       "L 80.795894 56.690976 \r\n",
       "L 82.589282 60.449347 \r\n",
       "L 84.38267 64.240897 \r\n",
       "L 86.176059 67.975396 \r\n",
       "L 87.969447 71.656059 \r\n",
       "L 89.762836 75.381701 \r\n",
       "L 91.556224 79.108211 \r\n",
       "L 93.349613 82.804104 \r\n",
       "L 95.143001 86.438694 \r\n",
       "L 96.936389 90.182344 \r\n",
       "L 98.729778 93.896076 \r\n",
       "L 100.523166 97.552231 \r\n",
       "L 102.316555 101.258362 \r\n",
       "L 104.109943 104.951407 \r\n",
       "L 105.903332 108.649037 \r\n",
       "L 107.69672 112.329434 \r\n",
       "L 109.490108 116.008428 \r\n",
       "L 111.283497 119.406165 \r\n",
       "L 113.076885 123.156393 \r\n",
       "L 114.870274 126.581951 \r\n",
       "L 116.663662 128.908687 \r\n",
       "L 118.457051 130.619469 \r\n",
       "L 120.250439 132.208744 \r\n",
       "L 122.043827 133.690314 \r\n",
       "L 123.837216 133.997559 \r\n",
       "L 125.630604 134.214002 \r\n",
       "L 127.423993 134.886841 \r\n",
       "L 129.217381 135.040201 \r\n",
       "L 131.01077 135.90456 \r\n",
       "L 132.804158 136.922727 \r\n",
       "L 134.597546 135.921231 \r\n",
       "L 136.390935 136.418491 \r\n",
       "L 138.184323 135.815406 \r\n",
       "L 139.977712 135.912106 \r\n",
       "L 141.7711 135.255957 \r\n",
       "L 143.564489 135.099513 \r\n",
       "L 145.357877 135.128535 \r\n",
       "L 147.151265 135.877812 \r\n",
       "L 148.944654 135.368871 \r\n",
       "L 150.738042 135.455012 \r\n",
       "L 152.531431 135.292478 \r\n",
       "L 154.324819 135.464626 \r\n",
       "L 156.118208 135.967377 \r\n",
       "L 157.911596 136.102204 \r\n",
       "L 159.704985 135.940862 \r\n",
       "L 161.498373 136.011019 \r\n",
       "L 163.291761 136.241783 \r\n",
       "L 165.08515 136.394871 \r\n",
       "L 166.878538 136.319987 \r\n",
       "L 168.671927 136.53362 \r\n",
       "L 170.465315 136.437707 \r\n",
       "L 172.258704 136.865495 \r\n",
       "L 174.052092 136.733143 \r\n",
       "L 175.84548 136.80094 \r\n",
       "L 177.638869 136.684989 \r\n",
       "L 179.432257 136.649891 \r\n",
       "L 181.225646 136.575371 \r\n",
       "L 183.019034 136.475724 \r\n",
       "L 184.812423 136.067279 \r\n",
       "L 186.605811 136.361226 \r\n",
       "L 188.399199 136.284037 \r\n",
       "L 190.192588 136.216036 \r\n",
       "L 191.985976 136.651954 \r\n",
       "L 193.779365 136.451127 \r\n",
       "L 195.572753 136.549735 \r\n",
       "L 197.366142 136.649405 \r\n",
       "L 199.15953 136.622805 \r\n",
       "L 200.952918 136.42204 \r\n",
       "L 202.746307 136.42204 \r\n",
       "L 204.539695 136.253402 \r\n",
       "L 206.333084 136.193753 \r\n",
       "L 208.126472 135.965181 \r\n",
       "L 209.919861 136.170465 \r\n",
       "L 211.713249 136.251536 \r\n",
       "L 213.506637 136.251536 \r\n",
       "L 215.300026 136.251536 \r\n",
       "L 217.093414 136.251536 \r\n",
       "L 218.886803 136.325349 \r\n",
       "L 220.680191 136.408489 \r\n",
       "L 222.47358 136.408946 \r\n",
       "L 224.266968 136.408946 \r\n",
       "L 226.060356 136.492596 \r\n",
       "L 227.853745 136.5235 \r\n",
       "L 229.647133 136.352944 \r\n",
       "L 231.440522 136.380166 \r\n",
       "L 233.23391 136.405066 \r\n",
       "L 235.027299 136.430124 \r\n",
       "L 236.820687 136.282323 \r\n",
       "L 238.614075 136.327138 \r\n",
       "L 240.407464 136.464767 \r\n",
       "L 242.200852 136.804534 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_13\">\r\n",
       "    <path clip-path=\"url(#p51a6f5e4d6)\" d=\"M 64.655398 13.377273 \r\n",
       "L 66.448786 13.471782 \r\n",
       "L 68.242175 13.502134 \r\n",
       "L 70.035563 13.519051 \r\n",
       "L 71.828951 13.527218 \r\n",
       "L 73.62234 13.53422 \r\n",
       "L 75.415728 13.537577 \r\n",
       "L 77.209117 13.539683 \r\n",
       "L 79.002505 13.54093 \r\n",
       "L 80.795894 13.541559 \r\n",
       "L 82.589282 13.542163 \r\n",
       "L 84.38267 13.542462 \r\n",
       "L 86.176059 13.542609 \r\n",
       "L 87.969447 13.542757 \r\n",
       "L 89.762836 13.54286 \r\n",
       "L 91.556224 13.542906 \r\n",
       "L 93.349613 13.542951 \r\n",
       "L 95.143001 13.542974 \r\n",
       "L 96.936389 13.542984 \r\n",
       "L 98.729778 13.542993 \r\n",
       "L 100.523166 13.542999 \r\n",
       "L 102.316555 13.543004 \r\n",
       "L 104.109943 13.543005 \r\n",
       "L 105.903332 13.543007 \r\n",
       "L 107.69672 13.543007 \r\n",
       "L 109.490108 13.543008 \r\n",
       "L 111.283497 13.543008 \r\n",
       "L 113.076885 13.543008 \r\n",
       "L 114.870274 13.543008 \r\n",
       "L 116.663662 13.543008 \r\n",
       "L 118.457051 13.543008 \r\n",
       "L 120.250439 13.543008 \r\n",
       "L 122.043827 13.543008 \r\n",
       "L 123.837216 13.543008 \r\n",
       "L 125.630604 13.543008 \r\n",
       "L 127.423993 13.543008 \r\n",
       "L 129.217381 13.543008 \r\n",
       "L 131.01077 13.543008 \r\n",
       "L 132.804158 13.543008 \r\n",
       "L 134.597546 13.543008 \r\n",
       "L 136.390935 13.543008 \r\n",
       "L 138.184323 13.543008 \r\n",
       "L 139.977712 13.543008 \r\n",
       "L 141.7711 13.543008 \r\n",
       "L 143.564489 13.543008 \r\n",
       "L 145.357877 13.543008 \r\n",
       "L 147.151265 13.543008 \r\n",
       "L 148.944654 13.543008 \r\n",
       "L 150.738042 13.543008 \r\n",
       "L 152.531431 13.543008 \r\n",
       "L 154.324819 13.543008 \r\n",
       "L 156.118208 13.543008 \r\n",
       "L 157.911596 13.543009 \r\n",
       "L 159.704985 13.543008 \r\n",
       "L 161.498373 13.543008 \r\n",
       "L 163.291761 13.543008 \r\n",
       "L 165.08515 13.543008 \r\n",
       "L 166.878538 13.543008 \r\n",
       "L 168.671927 13.543008 \r\n",
       "L 170.465315 13.543008 \r\n",
       "L 172.258704 13.543008 \r\n",
       "L 174.052092 13.543008 \r\n",
       "L 175.84548 13.543008 \r\n",
       "L 177.638869 13.543008 \r\n",
       "L 179.432257 13.543008 \r\n",
       "L 181.225646 13.543008 \r\n",
       "L 183.019034 13.543008 \r\n",
       "L 184.812423 13.543008 \r\n",
       "L 186.605811 13.543008 \r\n",
       "L 188.399199 13.543008 \r\n",
       "L 190.192588 13.543008 \r\n",
       "L 191.985976 13.543008 \r\n",
       "L 193.779365 13.543008 \r\n",
       "L 195.572753 13.543008 \r\n",
       "L 197.366142 13.543008 \r\n",
       "L 199.15953 13.543008 \r\n",
       "L 200.952918 13.543008 \r\n",
       "L 202.746307 13.543008 \r\n",
       "L 204.539695 13.543008 \r\n",
       "L 206.333084 13.543008 \r\n",
       "L 208.126472 13.543008 \r\n",
       "L 209.919861 13.543008 \r\n",
       "L 211.713249 13.543008 \r\n",
       "L 213.506637 13.543008 \r\n",
       "L 215.300026 13.543008 \r\n",
       "L 217.093414 13.543008 \r\n",
       "L 218.886803 13.543008 \r\n",
       "L 220.680191 13.543008 \r\n",
       "L 222.47358 13.543008 \r\n",
       "L 224.266968 13.543008 \r\n",
       "L 226.060356 13.543008 \r\n",
       "L 227.853745 13.543008 \r\n",
       "L 229.647133 13.543008 \r\n",
       "L 231.440522 13.543008 \r\n",
       "L 233.23391 13.543008 \r\n",
       "L 235.027299 13.543008 \r\n",
       "L 236.820687 13.543008 \r\n",
       "L 238.614075 13.543008 \r\n",
       "L 240.407464 13.543008 \r\n",
       "L 242.200852 13.543008 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 55.778125 143.1 \r\n",
       "L 55.778125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 251.078125 143.1 \r\n",
       "L 251.078125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 55.778125 143.1 \r\n",
       "L 251.078125 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 55.778125 7.2 \r\n",
       "L 251.078125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 188.803125 91.328125 \r\n",
       "L 244.078125 91.328125 \r\n",
       "Q 246.078125 91.328125 246.078125 89.328125 \r\n",
       "L 246.078125 60.971875 \r\n",
       "Q 246.078125 58.971875 244.078125 58.971875 \r\n",
       "L 188.803125 58.971875 \r\n",
       "Q 186.803125 58.971875 186.803125 60.971875 \r\n",
       "L 186.803125 89.328125 \r\n",
       "Q 186.803125 91.328125 188.803125 91.328125 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_14\">\r\n",
       "     <path d=\"M 190.803125 67.070312 \r\n",
       "L 210.803125 67.070312 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_15\"/>\r\n",
       "    <g id=\"text_14\">\r\n",
       "     <!-- train -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 18.3125 70.21875 \r\n",
       "L 18.3125 54.6875 \r\n",
       "L 36.8125 54.6875 \r\n",
       "L 36.8125 47.703125 \r\n",
       "L 18.3125 47.703125 \r\n",
       "L 18.3125 18.015625 \r\n",
       "Q 18.3125 11.328125 20.140625 9.421875 \r\n",
       "Q 21.96875 7.515625 27.59375 7.515625 \r\n",
       "L 36.8125 7.515625 \r\n",
       "L 36.8125 0 \r\n",
       "L 27.59375 0 \r\n",
       "Q 17.1875 0 13.234375 3.875 \r\n",
       "Q 9.28125 7.765625 9.28125 18.015625 \r\n",
       "L 9.28125 47.703125 \r\n",
       "L 2.6875 47.703125 \r\n",
       "L 2.6875 54.6875 \r\n",
       "L 9.28125 54.6875 \r\n",
       "L 9.28125 70.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-116\"/>\r\n",
       "      <path d=\"M 41.109375 46.296875 \r\n",
       "Q 39.59375 47.171875 37.8125 47.578125 \r\n",
       "Q 36.03125 48 33.890625 48 \r\n",
       "Q 26.265625 48 22.1875 43.046875 \r\n",
       "Q 18.109375 38.09375 18.109375 28.8125 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 20.953125 51.171875 25.484375 53.578125 \r\n",
       "Q 30.03125 56 36.53125 56 \r\n",
       "Q 37.453125 56 38.578125 55.875 \r\n",
       "Q 39.703125 55.765625 41.0625 55.515625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-114\"/>\r\n",
       "      <path d=\"M 34.28125 27.484375 \r\n",
       "Q 23.390625 27.484375 19.1875 25 \r\n",
       "Q 14.984375 22.515625 14.984375 16.5 \r\n",
       "Q 14.984375 11.71875 18.140625 8.90625 \r\n",
       "Q 21.296875 6.109375 26.703125 6.109375 \r\n",
       "Q 34.1875 6.109375 38.703125 11.40625 \r\n",
       "Q 43.21875 16.703125 43.21875 25.484375 \r\n",
       "L 43.21875 27.484375 \r\n",
       "z\r\n",
       "M 52.203125 31.203125 \r\n",
       "L 52.203125 0 \r\n",
       "L 43.21875 0 \r\n",
       "L 43.21875 8.296875 \r\n",
       "Q 40.140625 3.328125 35.546875 0.953125 \r\n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \r\n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \r\n",
       "Q 6 8.015625 6 15.921875 \r\n",
       "Q 6 25.140625 12.171875 29.828125 \r\n",
       "Q 18.359375 34.515625 30.609375 34.515625 \r\n",
       "L 43.21875 34.515625 \r\n",
       "L 43.21875 35.40625 \r\n",
       "Q 43.21875 41.609375 39.140625 45 \r\n",
       "Q 35.0625 48.390625 27.6875 48.390625 \r\n",
       "Q 23 48.390625 18.546875 47.265625 \r\n",
       "Q 14.109375 46.140625 10.015625 43.890625 \r\n",
       "L 10.015625 52.203125 \r\n",
       "Q 14.9375 54.109375 19.578125 55.046875 \r\n",
       "Q 24.21875 56 28.609375 56 \r\n",
       "Q 40.484375 56 46.34375 49.84375 \r\n",
       "Q 52.203125 43.703125 52.203125 31.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-97\"/>\r\n",
       "      <path d=\"M 9.421875 54.6875 \r\n",
       "L 18.40625 54.6875 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 64.59375 \r\n",
       "L 9.421875 64.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-105\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-110\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(218.803125 70.570312)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_16\">\r\n",
       "     <path d=\"M 190.803125 81.748437 \r\n",
       "L 210.803125 81.748437 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_17\"/>\r\n",
       "    <g id=\"text_15\">\r\n",
       "     <!-- test -->\r\n",
       "     <g transform=\"translate(218.803125 85.248437)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p51a6f5e4d6\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"55.778125\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_and_plot(lambd=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_ejl383l",
    "id": "F2335EC469AB402C890CE258932583D1",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 使用权重衰减"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "graffitiCellId": "id_x1tkbn7",
    "id": "0770D8C23B8144C59D13D24390E471F0",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L2 norm of w: 0.034862667322158813\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 254.544602 180.65625\" width=\"254.544602pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 254.544602 180.65625 \r\n",
       "L 254.544602 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 51.378125 143.1 \r\n",
       "L 246.678125 143.1 \r\n",
       "L 246.678125 7.2 \r\n",
       "L 51.378125 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"me670b43922\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"58.462009\" xlink:href=\"#me670b43922\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-48\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(55.280759 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"94.329778\" xlink:href=\"#me670b43922\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 20 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-50\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(87.967278 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"130.197546\" xlink:href=\"#me670b43922\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 40 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-52\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(123.835046 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"166.065315\" xlink:href=\"#me670b43922\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 60 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 33.015625 40.375 \r\n",
       "Q 26.375 40.375 22.484375 35.828125 \r\n",
       "Q 18.609375 31.296875 18.609375 23.390625 \r\n",
       "Q 18.609375 15.53125 22.484375 10.953125 \r\n",
       "Q 26.375 6.390625 33.015625 6.390625 \r\n",
       "Q 39.65625 6.390625 43.53125 10.953125 \r\n",
       "Q 47.40625 15.53125 47.40625 23.390625 \r\n",
       "Q 47.40625 31.296875 43.53125 35.828125 \r\n",
       "Q 39.65625 40.375 33.015625 40.375 \r\n",
       "z\r\n",
       "M 52.59375 71.296875 \r\n",
       "L 52.59375 62.3125 \r\n",
       "Q 48.875 64.0625 45.09375 64.984375 \r\n",
       "Q 41.3125 65.921875 37.59375 65.921875 \r\n",
       "Q 27.828125 65.921875 22.671875 59.328125 \r\n",
       "Q 17.53125 52.734375 16.796875 39.40625 \r\n",
       "Q 19.671875 43.65625 24.015625 45.921875 \r\n",
       "Q 28.375 48.1875 33.59375 48.1875 \r\n",
       "Q 44.578125 48.1875 50.953125 41.515625 \r\n",
       "Q 57.328125 34.859375 57.328125 23.390625 \r\n",
       "Q 57.328125 12.15625 50.6875 5.359375 \r\n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \r\n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \r\n",
       "Q 6.984375 17.96875 6.984375 36.375 \r\n",
       "Q 6.984375 53.65625 15.1875 63.9375 \r\n",
       "Q 23.390625 74.21875 37.203125 74.21875 \r\n",
       "Q 40.921875 74.21875 44.703125 73.484375 \r\n",
       "Q 48.484375 72.75 52.59375 71.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-54\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(159.702815 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-54\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"201.933084\" xlink:href=\"#me670b43922\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 80 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 34.625 \r\n",
       "Q 24.75 34.625 20.71875 30.859375 \r\n",
       "Q 16.703125 27.09375 16.703125 20.515625 \r\n",
       "Q 16.703125 13.921875 20.71875 10.15625 \r\n",
       "Q 24.75 6.390625 31.78125 6.390625 \r\n",
       "Q 38.8125 6.390625 42.859375 10.171875 \r\n",
       "Q 46.921875 13.96875 46.921875 20.515625 \r\n",
       "Q 46.921875 27.09375 42.890625 30.859375 \r\n",
       "Q 38.875 34.625 31.78125 34.625 \r\n",
       "z\r\n",
       "M 21.921875 38.8125 \r\n",
       "Q 15.578125 40.375 12.03125 44.71875 \r\n",
       "Q 8.5 49.078125 8.5 55.328125 \r\n",
       "Q 8.5 64.0625 14.71875 69.140625 \r\n",
       "Q 20.953125 74.21875 31.78125 74.21875 \r\n",
       "Q 42.671875 74.21875 48.875 69.140625 \r\n",
       "Q 55.078125 64.0625 55.078125 55.328125 \r\n",
       "Q 55.078125 49.078125 51.53125 44.71875 \r\n",
       "Q 48 40.375 41.703125 38.8125 \r\n",
       "Q 48.828125 37.15625 52.796875 32.3125 \r\n",
       "Q 56.78125 27.484375 56.78125 20.515625 \r\n",
       "Q 56.78125 9.90625 50.3125 4.234375 \r\n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.734375 -1.421875 13.25 4.234375 \r\n",
       "Q 6.78125 9.90625 6.78125 20.515625 \r\n",
       "Q 6.78125 27.484375 10.78125 32.3125 \r\n",
       "Q 14.796875 37.15625 21.921875 38.8125 \r\n",
       "z\r\n",
       "M 18.3125 54.390625 \r\n",
       "Q 18.3125 48.734375 21.84375 45.5625 \r\n",
       "Q 25.390625 42.390625 31.78125 42.390625 \r\n",
       "Q 38.140625 42.390625 41.71875 45.5625 \r\n",
       "Q 45.3125 48.734375 45.3125 54.390625 \r\n",
       "Q 45.3125 60.0625 41.71875 63.234375 \r\n",
       "Q 38.140625 66.40625 31.78125 66.40625 \r\n",
       "Q 25.390625 66.40625 21.84375 63.234375 \r\n",
       "Q 18.3125 60.0625 18.3125 54.390625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-56\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(195.570584 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-56\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"237.800852\" xlink:href=\"#me670b43922\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-49\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(228.257102 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_7\">\r\n",
       "     <!-- epochs -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 56.203125 29.59375 \r\n",
       "L 56.203125 25.203125 \r\n",
       "L 14.890625 25.203125 \r\n",
       "Q 15.484375 15.921875 20.484375 11.0625 \r\n",
       "Q 25.484375 6.203125 34.421875 6.203125 \r\n",
       "Q 39.59375 6.203125 44.453125 7.46875 \r\n",
       "Q 49.3125 8.734375 54.109375 11.28125 \r\n",
       "L 54.109375 2.78125 \r\n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \r\n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \r\n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \r\n",
       "Q 5.515625 13.8125 5.515625 26.8125 \r\n",
       "Q 5.515625 40.234375 12.765625 48.109375 \r\n",
       "Q 20.015625 56 32.328125 56 \r\n",
       "Q 43.359375 56 49.78125 48.890625 \r\n",
       "Q 56.203125 41.796875 56.203125 29.59375 \r\n",
       "z\r\n",
       "M 47.21875 32.234375 \r\n",
       "Q 47.125 39.59375 43.09375 43.984375 \r\n",
       "Q 39.0625 48.390625 32.421875 48.390625 \r\n",
       "Q 24.90625 48.390625 20.390625 44.140625 \r\n",
       "Q 15.875 39.890625 15.1875 32.171875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-101\"/>\r\n",
       "      <path d=\"M 18.109375 8.203125 \r\n",
       "L 18.109375 -20.796875 \r\n",
       "L 9.078125 -20.796875 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.390625 \r\n",
       "Q 20.953125 51.265625 25.265625 53.625 \r\n",
       "Q 29.59375 56 35.59375 56 \r\n",
       "Q 45.5625 56 51.78125 48.09375 \r\n",
       "Q 58.015625 40.1875 58.015625 27.296875 \r\n",
       "Q 58.015625 14.40625 51.78125 6.484375 \r\n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \r\n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \r\n",
       "Q 20.953125 3.328125 18.109375 8.203125 \r\n",
       "z\r\n",
       "M 48.6875 27.296875 \r\n",
       "Q 48.6875 37.203125 44.609375 42.84375 \r\n",
       "Q 40.53125 48.484375 33.40625 48.484375 \r\n",
       "Q 26.265625 48.484375 22.1875 42.84375 \r\n",
       "Q 18.109375 37.203125 18.109375 27.296875 \r\n",
       "Q 18.109375 17.390625 22.1875 11.75 \r\n",
       "Q 26.265625 6.109375 33.40625 6.109375 \r\n",
       "Q 40.53125 6.109375 44.609375 11.75 \r\n",
       "Q 48.6875 17.390625 48.6875 27.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-112\"/>\r\n",
       "      <path d=\"M 30.609375 48.390625 \r\n",
       "Q 23.390625 48.390625 19.1875 42.75 \r\n",
       "Q 14.984375 37.109375 14.984375 27.296875 \r\n",
       "Q 14.984375 17.484375 19.15625 11.84375 \r\n",
       "Q 23.34375 6.203125 30.609375 6.203125 \r\n",
       "Q 37.796875 6.203125 41.984375 11.859375 \r\n",
       "Q 46.1875 17.53125 46.1875 27.296875 \r\n",
       "Q 46.1875 37.015625 41.984375 42.703125 \r\n",
       "Q 37.796875 48.390625 30.609375 48.390625 \r\n",
       "z\r\n",
       "M 30.609375 56 \r\n",
       "Q 42.328125 56 49.015625 48.375 \r\n",
       "Q 55.71875 40.765625 55.71875 27.296875 \r\n",
       "Q 55.71875 13.875 49.015625 6.21875 \r\n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \r\n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \r\n",
       "Q 5.515625 13.875 5.515625 27.296875 \r\n",
       "Q 5.515625 40.765625 12.171875 48.375 \r\n",
       "Q 18.84375 56 30.609375 56 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-111\"/>\r\n",
       "      <path d=\"M 48.78125 52.59375 \r\n",
       "L 48.78125 44.1875 \r\n",
       "Q 44.96875 46.296875 41.140625 47.34375 \r\n",
       "Q 37.3125 48.390625 33.40625 48.390625 \r\n",
       "Q 24.65625 48.390625 19.8125 42.84375 \r\n",
       "Q 14.984375 37.3125 14.984375 27.296875 \r\n",
       "Q 14.984375 17.28125 19.8125 11.734375 \r\n",
       "Q 24.65625 6.203125 33.40625 6.203125 \r\n",
       "Q 37.3125 6.203125 41.140625 7.25 \r\n",
       "Q 44.96875 8.296875 48.78125 10.40625 \r\n",
       "L 48.78125 2.09375 \r\n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \r\n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \r\n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \r\n",
       "Q 5.515625 14.109375 5.515625 27.296875 \r\n",
       "Q 5.515625 40.671875 12.859375 48.328125 \r\n",
       "Q 20.21875 56 33.015625 56 \r\n",
       "Q 37.15625 56 41.109375 55.140625 \r\n",
       "Q 45.0625 54.296875 48.78125 52.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-99\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 75.984375 \r\n",
       "L 18.109375 75.984375 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-104\"/>\r\n",
       "      <path d=\"M 44.28125 53.078125 \r\n",
       "L 44.28125 44.578125 \r\n",
       "Q 40.484375 46.53125 36.375 47.5 \r\n",
       "Q 32.28125 48.484375 27.875 48.484375 \r\n",
       "Q 21.1875 48.484375 17.84375 46.4375 \r\n",
       "Q 14.5 44.390625 14.5 40.28125 \r\n",
       "Q 14.5 37.15625 16.890625 35.375 \r\n",
       "Q 19.28125 33.59375 26.515625 31.984375 \r\n",
       "L 29.59375 31.296875 \r\n",
       "Q 39.15625 29.25 43.1875 25.515625 \r\n",
       "Q 47.21875 21.78125 47.21875 15.09375 \r\n",
       "Q 47.21875 7.46875 41.1875 3.015625 \r\n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \r\n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \r\n",
       "Q 10.6875 0.296875 5.421875 2 \r\n",
       "L 5.421875 11.28125 \r\n",
       "Q 10.40625 8.6875 15.234375 7.390625 \r\n",
       "Q 20.0625 6.109375 24.8125 6.109375 \r\n",
       "Q 31.15625 6.109375 34.5625 8.28125 \r\n",
       "Q 37.984375 10.453125 37.984375 14.40625 \r\n",
       "Q 37.984375 18.0625 35.515625 20.015625 \r\n",
       "Q 33.0625 21.96875 24.703125 23.78125 \r\n",
       "L 21.578125 24.515625 \r\n",
       "Q 13.234375 26.265625 9.515625 29.90625 \r\n",
       "Q 5.8125 33.546875 5.8125 39.890625 \r\n",
       "Q 5.8125 47.609375 11.28125 51.796875 \r\n",
       "Q 16.75 56 26.8125 56 \r\n",
       "Q 31.78125 56 36.171875 55.265625 \r\n",
       "Q 40.578125 54.546875 44.28125 53.078125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-115\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\r\n",
       "      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"mff9576d1ce\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mff9576d1ce\" y=\"101.351725\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- $\\mathdefault{10^{-2}}$ -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-8722\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 105.150944)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mff9576d1ce\" y=\"57.003842\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- $\\mathdefault{10^{0}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 60.803061)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mff9576d1ce\" y=\"12.655958\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- $\\mathdefault{10^{2}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 16.455177)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_11\">\r\n",
       "     <!-- loss -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-108\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-108\"/>\r\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_10\">\r\n",
       "    <path clip-path=\"url(#p71afd1fdb7)\" d=\"M 60.255398 37.594153 \r\n",
       "L 62.048786 56.99161 \r\n",
       "L 63.842175 73.59599 \r\n",
       "L 65.635563 88.290509 \r\n",
       "L 67.428951 101.993367 \r\n",
       "L 69.22234 114.338296 \r\n",
       "L 71.015728 123.780611 \r\n",
       "L 72.809117 130.316689 \r\n",
       "L 74.602505 132.170538 \r\n",
       "L 76.395894 135.511396 \r\n",
       "L 78.189282 134.225451 \r\n",
       "L 79.98267 134.532628 \r\n",
       "L 81.776059 135.188698 \r\n",
       "L 83.569447 135.791271 \r\n",
       "L 85.362836 135.83013 \r\n",
       "L 87.156224 136.304213 \r\n",
       "L 88.949613 136.340802 \r\n",
       "L 90.743001 135.197927 \r\n",
       "L 92.536389 134.44789 \r\n",
       "L 94.329778 135.957779 \r\n",
       "L 96.123166 134.002561 \r\n",
       "L 97.916555 136.577596 \r\n",
       "L 99.709943 134.577511 \r\n",
       "L 101.503332 134.472974 \r\n",
       "L 103.29672 134.451785 \r\n",
       "L 105.090108 134.193294 \r\n",
       "L 106.883497 135.864301 \r\n",
       "L 108.676885 135.07328 \r\n",
       "L 110.470274 133.989632 \r\n",
       "L 112.263662 135.958327 \r\n",
       "L 114.057051 134.375466 \r\n",
       "L 115.850439 134.75087 \r\n",
       "L 117.643827 134.418734 \r\n",
       "L 119.437216 136.664202 \r\n",
       "L 121.230604 135.561424 \r\n",
       "L 123.023993 135.4763 \r\n",
       "L 124.817381 134.463975 \r\n",
       "L 126.61077 135.119161 \r\n",
       "L 128.404158 135.754886 \r\n",
       "L 130.197546 136.288657 \r\n",
       "L 131.990935 134.926704 \r\n",
       "L 133.784323 136.201497 \r\n",
       "L 135.577712 136.031983 \r\n",
       "L 137.3711 135.758191 \r\n",
       "L 139.164489 135.222813 \r\n",
       "L 140.957877 135.656806 \r\n",
       "L 142.751265 135.807812 \r\n",
       "L 144.544654 135.959965 \r\n",
       "L 146.338042 135.751003 \r\n",
       "L 148.131431 135.68576 \r\n",
       "L 149.924819 136.675931 \r\n",
       "L 151.718208 134.380556 \r\n",
       "L 153.511596 133.714201 \r\n",
       "L 155.304985 135.180656 \r\n",
       "L 157.098373 136.084332 \r\n",
       "L 158.891761 135.579628 \r\n",
       "L 160.68515 136.646394 \r\n",
       "L 162.478538 135.319968 \r\n",
       "L 164.271927 136.663821 \r\n",
       "L 166.065315 135.740979 \r\n",
       "L 167.858704 136.475149 \r\n",
       "L 169.652092 134.615078 \r\n",
       "L 171.44548 134.998433 \r\n",
       "L 173.238869 135.547736 \r\n",
       "L 175.032257 136.098285 \r\n",
       "L 176.825646 134.998661 \r\n",
       "L 178.619034 135.060351 \r\n",
       "L 180.412423 134.443455 \r\n",
       "L 182.205811 136.298374 \r\n",
       "L 183.999199 135.442881 \r\n",
       "L 185.792588 135.885167 \r\n",
       "L 187.585976 135.62263 \r\n",
       "L 189.379365 135.324748 \r\n",
       "L 191.172753 135.729437 \r\n",
       "L 192.966142 135.245912 \r\n",
       "L 194.75953 136.922727 \r\n",
       "L 196.552918 135.859597 \r\n",
       "L 198.346307 134.6122 \r\n",
       "L 200.139695 134.378344 \r\n",
       "L 201.933084 133.85516 \r\n",
       "L 203.726472 134.073113 \r\n",
       "L 205.519861 134.504108 \r\n",
       "L 207.313249 134.277971 \r\n",
       "L 209.106637 134.533646 \r\n",
       "L 210.900026 134.975934 \r\n",
       "L 212.693414 134.358651 \r\n",
       "L 214.486803 134.198745 \r\n",
       "L 216.280191 135.254348 \r\n",
       "L 218.07358 135.388743 \r\n",
       "L 219.866968 134.939165 \r\n",
       "L 221.660356 136.293653 \r\n",
       "L 223.453745 134.227051 \r\n",
       "L 225.247133 135.827574 \r\n",
       "L 227.040522 135.869693 \r\n",
       "L 228.83391 136.443086 \r\n",
       "L 230.627299 135.184068 \r\n",
       "L 232.420687 136.685473 \r\n",
       "L 234.214075 135.576678 \r\n",
       "L 236.007464 134.229225 \r\n",
       "L 237.800852 136.873675 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_11\">\r\n",
       "    <path clip-path=\"url(#p71afd1fdb7)\" d=\"M 60.255398 13.377273 \r\n",
       "L 62.048786 16.756649 \r\n",
       "L 63.842175 20.169436 \r\n",
       "L 65.635563 23.613533 \r\n",
       "L 67.428951 27.072513 \r\n",
       "L 69.22234 30.536831 \r\n",
       "L 71.015728 34.005322 \r\n",
       "L 72.809117 37.472333 \r\n",
       "L 74.602505 40.9362 \r\n",
       "L 76.395894 44.400458 \r\n",
       "L 78.189282 47.857206 \r\n",
       "L 79.98267 51.306788 \r\n",
       "L 81.776059 54.74823 \r\n",
       "L 83.569447 58.181139 \r\n",
       "L 85.362836 61.598985 \r\n",
       "L 87.156224 64.998272 \r\n",
       "L 88.949613 68.36745 \r\n",
       "L 90.743001 71.689822 \r\n",
       "L 92.536389 74.972955 \r\n",
       "L 94.329778 78.197172 \r\n",
       "L 96.123166 81.328604 \r\n",
       "L 97.916555 84.362452 \r\n",
       "L 99.709943 87.250351 \r\n",
       "L 101.503332 89.972557 \r\n",
       "L 103.29672 92.466955 \r\n",
       "L 105.090108 94.755725 \r\n",
       "L 106.883497 96.743372 \r\n",
       "L 108.676885 98.453505 \r\n",
       "L 110.470274 99.88944 \r\n",
       "L 112.263662 101.064227 \r\n",
       "L 114.057051 101.991739 \r\n",
       "L 115.850439 102.737704 \r\n",
       "L 117.643827 103.312272 \r\n",
       "L 119.437216 103.745814 \r\n",
       "L 121.230604 104.068018 \r\n",
       "L 123.023993 104.303452 \r\n",
       "L 124.817381 104.481192 \r\n",
       "L 126.61077 104.631529 \r\n",
       "L 128.404158 104.727582 \r\n",
       "L 130.197546 104.826404 \r\n",
       "L 131.990935 104.907701 \r\n",
       "L 133.784323 104.963771 \r\n",
       "L 135.577712 104.991962 \r\n",
       "L 137.3711 104.991742 \r\n",
       "L 139.164489 105.024255 \r\n",
       "L 140.957877 105.038758 \r\n",
       "L 142.751265 105.067821 \r\n",
       "L 144.544654 105.101172 \r\n",
       "L 146.338042 105.081609 \r\n",
       "L 148.131431 105.063515 \r\n",
       "L 149.924819 105.058915 \r\n",
       "L 151.718208 105.07958 \r\n",
       "L 153.511596 105.090869 \r\n",
       "L 155.304985 105.0915 \r\n",
       "L 157.098373 105.086283 \r\n",
       "L 158.891761 105.094031 \r\n",
       "L 160.68515 105.0767 \r\n",
       "L 162.478538 105.095648 \r\n",
       "L 164.271927 105.078306 \r\n",
       "L 166.065315 105.089086 \r\n",
       "L 167.858704 105.082125 \r\n",
       "L 169.652092 105.088261 \r\n",
       "L 171.44548 105.075161 \r\n",
       "L 173.238869 105.074289 \r\n",
       "L 175.032257 105.063568 \r\n",
       "L 176.825646 105.102814 \r\n",
       "L 178.619034 105.075137 \r\n",
       "L 180.412423 105.096314 \r\n",
       "L 182.205811 105.096139 \r\n",
       "L 183.999199 105.070325 \r\n",
       "L 185.792588 105.05753 \r\n",
       "L 187.585976 105.042986 \r\n",
       "L 189.379365 105.049961 \r\n",
       "L 191.172753 105.058961 \r\n",
       "L 192.966142 105.040749 \r\n",
       "L 194.75953 105.041264 \r\n",
       "L 196.552918 105.054733 \r\n",
       "L 198.346307 105.045485 \r\n",
       "L 200.139695 105.052414 \r\n",
       "L 201.933084 105.055528 \r\n",
       "L 203.726472 105.047931 \r\n",
       "L 205.519861 105.06416 \r\n",
       "L 207.313249 105.053548 \r\n",
       "L 209.106637 105.045513 \r\n",
       "L 210.900026 105.035639 \r\n",
       "L 212.693414 105.053304 \r\n",
       "L 214.486803 105.062486 \r\n",
       "L 216.280191 105.037541 \r\n",
       "L 218.07358 105.042394 \r\n",
       "L 219.866968 105.038453 \r\n",
       "L 221.660356 105.024462 \r\n",
       "L 223.453745 105.016028 \r\n",
       "L 225.247133 104.994383 \r\n",
       "L 227.040522 105.010637 \r\n",
       "L 228.83391 105.001882 \r\n",
       "L 230.627299 105.008305 \r\n",
       "L 232.420687 105.016456 \r\n",
       "L 234.214075 105.006909 \r\n",
       "L 236.007464 105.008359 \r\n",
       "L 237.800852 104.989098 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 51.378125 143.1 \r\n",
       "L 51.378125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 246.678125 143.1 \r\n",
       "L 246.678125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 51.378125 143.1 \r\n",
       "L 246.678125 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 51.378125 7.2 \r\n",
       "L 246.678125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 184.403125 44.55625 \r\n",
       "L 239.678125 44.55625 \r\n",
       "Q 241.678125 44.55625 241.678125 42.55625 \r\n",
       "L 241.678125 14.2 \r\n",
       "Q 241.678125 12.2 239.678125 12.2 \r\n",
       "L 184.403125 12.2 \r\n",
       "Q 182.403125 12.2 182.403125 14.2 \r\n",
       "L 182.403125 42.55625 \r\n",
       "Q 182.403125 44.55625 184.403125 44.55625 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_12\">\r\n",
       "     <path d=\"M 186.403125 20.298437 \r\n",
       "L 206.403125 20.298437 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_13\"/>\r\n",
       "    <g id=\"text_12\">\r\n",
       "     <!-- train -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 18.3125 70.21875 \r\n",
       "L 18.3125 54.6875 \r\n",
       "L 36.8125 54.6875 \r\n",
       "L 36.8125 47.703125 \r\n",
       "L 18.3125 47.703125 \r\n",
       "L 18.3125 18.015625 \r\n",
       "Q 18.3125 11.328125 20.140625 9.421875 \r\n",
       "Q 21.96875 7.515625 27.59375 7.515625 \r\n",
       "L 36.8125 7.515625 \r\n",
       "L 36.8125 0 \r\n",
       "L 27.59375 0 \r\n",
       "Q 17.1875 0 13.234375 3.875 \r\n",
       "Q 9.28125 7.765625 9.28125 18.015625 \r\n",
       "L 9.28125 47.703125 \r\n",
       "L 2.6875 47.703125 \r\n",
       "L 2.6875 54.6875 \r\n",
       "L 9.28125 54.6875 \r\n",
       "L 9.28125 70.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-116\"/>\r\n",
       "      <path d=\"M 41.109375 46.296875 \r\n",
       "Q 39.59375 47.171875 37.8125 47.578125 \r\n",
       "Q 36.03125 48 33.890625 48 \r\n",
       "Q 26.265625 48 22.1875 43.046875 \r\n",
       "Q 18.109375 38.09375 18.109375 28.8125 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 20.953125 51.171875 25.484375 53.578125 \r\n",
       "Q 30.03125 56 36.53125 56 \r\n",
       "Q 37.453125 56 38.578125 55.875 \r\n",
       "Q 39.703125 55.765625 41.0625 55.515625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-114\"/>\r\n",
       "      <path d=\"M 34.28125 27.484375 \r\n",
       "Q 23.390625 27.484375 19.1875 25 \r\n",
       "Q 14.984375 22.515625 14.984375 16.5 \r\n",
       "Q 14.984375 11.71875 18.140625 8.90625 \r\n",
       "Q 21.296875 6.109375 26.703125 6.109375 \r\n",
       "Q 34.1875 6.109375 38.703125 11.40625 \r\n",
       "Q 43.21875 16.703125 43.21875 25.484375 \r\n",
       "L 43.21875 27.484375 \r\n",
       "z\r\n",
       "M 52.203125 31.203125 \r\n",
       "L 52.203125 0 \r\n",
       "L 43.21875 0 \r\n",
       "L 43.21875 8.296875 \r\n",
       "Q 40.140625 3.328125 35.546875 0.953125 \r\n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \r\n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \r\n",
       "Q 6 8.015625 6 15.921875 \r\n",
       "Q 6 25.140625 12.171875 29.828125 \r\n",
       "Q 18.359375 34.515625 30.609375 34.515625 \r\n",
       "L 43.21875 34.515625 \r\n",
       "L 43.21875 35.40625 \r\n",
       "Q 43.21875 41.609375 39.140625 45 \r\n",
       "Q 35.0625 48.390625 27.6875 48.390625 \r\n",
       "Q 23 48.390625 18.546875 47.265625 \r\n",
       "Q 14.109375 46.140625 10.015625 43.890625 \r\n",
       "L 10.015625 52.203125 \r\n",
       "Q 14.9375 54.109375 19.578125 55.046875 \r\n",
       "Q 24.21875 56 28.609375 56 \r\n",
       "Q 40.484375 56 46.34375 49.84375 \r\n",
       "Q 52.203125 43.703125 52.203125 31.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-97\"/>\r\n",
       "      <path d=\"M 9.421875 54.6875 \r\n",
       "L 18.40625 54.6875 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 64.59375 \r\n",
       "L 9.421875 64.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-105\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-110\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(214.403125 23.798437)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_14\">\r\n",
       "     <path d=\"M 186.403125 34.976562 \r\n",
       "L 206.403125 34.976562 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_15\"/>\r\n",
       "    <g id=\"text_13\">\r\n",
       "     <!-- test -->\r\n",
       "     <g transform=\"translate(214.403125 38.476562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p71afd1fdb7\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"51.378125\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_and_plot(lambd=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_g2d3ly2",
    "id": "FD1464A14A6149AB81CF443A10F843A4",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 简洁实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "graffitiCellId": "id_b6kxfkc",
    "id": "20CFA1339D054931892561E254150368",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def fit_and_plot_pytorch(wd):\n",
    "    # 对权重参数衰减。权重名称一般是以weight结尾\n",
    "    net = nn.Linear(num_inputs, 1)\n",
    "    nn.init.normal_(net.weight, mean=0, std=1)\n",
    "    nn.init.normal_(net.bias, mean=0, std=1)\n",
    "    optimizer_w = torch.optim.SGD(params=[net.weight], lr=lr, weight_decay=wd) # 对权重参数衰减， weight_decay是表明是否启用权重衰减\n",
    "    optimizer_b = torch.optim.SGD(params=[net.bias], lr=lr)  # 不对偏差参数衰减\n",
    "    \n",
    "    train_ls, test_ls = [], []\n",
    "    for _ in range(num_epochs):\n",
    "        for X, y in train_iter:\n",
    "            l = loss(net(X), y).mean()\n",
    "            optimizer_w.zero_grad()\n",
    "            optimizer_b.zero_grad()\n",
    "            \n",
    "            l.backward()\n",
    "            \n",
    "            # 对两个optimizer实例分别调用step函数，从而分别更新权重和偏差\n",
    "            optimizer_w.step()\n",
    "            optimizer_b.step()\n",
    "        train_ls.append(loss(net(train_features), train_labels).mean().item())\n",
    "        test_ls.append(loss(net(test_features), test_labels).mean().item())\n",
    "    d2l.semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',\n",
    "                 range(1, num_epochs + 1), test_ls, ['train', 'test'])\n",
    "    print('L2 norm of w:', net.weight.data.norm().item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "graffitiCellId": "id_dxnud8x",
    "id": "525D01167F0E40509495588D6B0A0FB9",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L2 norm of w: 15.078595161437988\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 258.944602 180.65625\" width=\"258.944602pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 258.944602 180.65625 \r\n",
       "L 258.944602 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 55.778125 143.1 \r\n",
       "L 251.078125 143.1 \r\n",
       "L 251.078125 7.2 \r\n",
       "L 55.778125 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"md758e72895\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"62.862009\" xlink:href=\"#md758e72895\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-48\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(59.680759 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"98.729778\" xlink:href=\"#md758e72895\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 20 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-50\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(92.367278 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"134.597546\" xlink:href=\"#md758e72895\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 40 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-52\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(128.235046 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"170.465315\" xlink:href=\"#md758e72895\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 60 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 33.015625 40.375 \r\n",
       "Q 26.375 40.375 22.484375 35.828125 \r\n",
       "Q 18.609375 31.296875 18.609375 23.390625 \r\n",
       "Q 18.609375 15.53125 22.484375 10.953125 \r\n",
       "Q 26.375 6.390625 33.015625 6.390625 \r\n",
       "Q 39.65625 6.390625 43.53125 10.953125 \r\n",
       "Q 47.40625 15.53125 47.40625 23.390625 \r\n",
       "Q 47.40625 31.296875 43.53125 35.828125 \r\n",
       "Q 39.65625 40.375 33.015625 40.375 \r\n",
       "z\r\n",
       "M 52.59375 71.296875 \r\n",
       "L 52.59375 62.3125 \r\n",
       "Q 48.875 64.0625 45.09375 64.984375 \r\n",
       "Q 41.3125 65.921875 37.59375 65.921875 \r\n",
       "Q 27.828125 65.921875 22.671875 59.328125 \r\n",
       "Q 17.53125 52.734375 16.796875 39.40625 \r\n",
       "Q 19.671875 43.65625 24.015625 45.921875 \r\n",
       "Q 28.375 48.1875 33.59375 48.1875 \r\n",
       "Q 44.578125 48.1875 50.953125 41.515625 \r\n",
       "Q 57.328125 34.859375 57.328125 23.390625 \r\n",
       "Q 57.328125 12.15625 50.6875 5.359375 \r\n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \r\n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \r\n",
       "Q 6.984375 17.96875 6.984375 36.375 \r\n",
       "Q 6.984375 53.65625 15.1875 63.9375 \r\n",
       "Q 23.390625 74.21875 37.203125 74.21875 \r\n",
       "Q 40.921875 74.21875 44.703125 73.484375 \r\n",
       "Q 48.484375 72.75 52.59375 71.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-54\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(164.102815 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-54\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"206.333084\" xlink:href=\"#md758e72895\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 80 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 34.625 \r\n",
       "Q 24.75 34.625 20.71875 30.859375 \r\n",
       "Q 16.703125 27.09375 16.703125 20.515625 \r\n",
       "Q 16.703125 13.921875 20.71875 10.15625 \r\n",
       "Q 24.75 6.390625 31.78125 6.390625 \r\n",
       "Q 38.8125 6.390625 42.859375 10.171875 \r\n",
       "Q 46.921875 13.96875 46.921875 20.515625 \r\n",
       "Q 46.921875 27.09375 42.890625 30.859375 \r\n",
       "Q 38.875 34.625 31.78125 34.625 \r\n",
       "z\r\n",
       "M 21.921875 38.8125 \r\n",
       "Q 15.578125 40.375 12.03125 44.71875 \r\n",
       "Q 8.5 49.078125 8.5 55.328125 \r\n",
       "Q 8.5 64.0625 14.71875 69.140625 \r\n",
       "Q 20.953125 74.21875 31.78125 74.21875 \r\n",
       "Q 42.671875 74.21875 48.875 69.140625 \r\n",
       "Q 55.078125 64.0625 55.078125 55.328125 \r\n",
       "Q 55.078125 49.078125 51.53125 44.71875 \r\n",
       "Q 48 40.375 41.703125 38.8125 \r\n",
       "Q 48.828125 37.15625 52.796875 32.3125 \r\n",
       "Q 56.78125 27.484375 56.78125 20.515625 \r\n",
       "Q 56.78125 9.90625 50.3125 4.234375 \r\n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.734375 -1.421875 13.25 4.234375 \r\n",
       "Q 6.78125 9.90625 6.78125 20.515625 \r\n",
       "Q 6.78125 27.484375 10.78125 32.3125 \r\n",
       "Q 14.796875 37.15625 21.921875 38.8125 \r\n",
       "z\r\n",
       "M 18.3125 54.390625 \r\n",
       "Q 18.3125 48.734375 21.84375 45.5625 \r\n",
       "Q 25.390625 42.390625 31.78125 42.390625 \r\n",
       "Q 38.140625 42.390625 41.71875 45.5625 \r\n",
       "Q 45.3125 48.734375 45.3125 54.390625 \r\n",
       "Q 45.3125 60.0625 41.71875 63.234375 \r\n",
       "Q 38.140625 66.40625 31.78125 66.40625 \r\n",
       "Q 25.390625 66.40625 21.84375 63.234375 \r\n",
       "Q 18.3125 60.0625 18.3125 54.390625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-56\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(199.970584 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-56\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"242.200852\" xlink:href=\"#md758e72895\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-49\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(232.657102 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_7\">\r\n",
       "     <!-- epochs -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 56.203125 29.59375 \r\n",
       "L 56.203125 25.203125 \r\n",
       "L 14.890625 25.203125 \r\n",
       "Q 15.484375 15.921875 20.484375 11.0625 \r\n",
       "Q 25.484375 6.203125 34.421875 6.203125 \r\n",
       "Q 39.59375 6.203125 44.453125 7.46875 \r\n",
       "Q 49.3125 8.734375 54.109375 11.28125 \r\n",
       "L 54.109375 2.78125 \r\n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \r\n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \r\n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \r\n",
       "Q 5.515625 13.8125 5.515625 26.8125 \r\n",
       "Q 5.515625 40.234375 12.765625 48.109375 \r\n",
       "Q 20.015625 56 32.328125 56 \r\n",
       "Q 43.359375 56 49.78125 48.890625 \r\n",
       "Q 56.203125 41.796875 56.203125 29.59375 \r\n",
       "z\r\n",
       "M 47.21875 32.234375 \r\n",
       "Q 47.125 39.59375 43.09375 43.984375 \r\n",
       "Q 39.0625 48.390625 32.421875 48.390625 \r\n",
       "Q 24.90625 48.390625 20.390625 44.140625 \r\n",
       "Q 15.875 39.890625 15.1875 32.171875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-101\"/>\r\n",
       "      <path d=\"M 18.109375 8.203125 \r\n",
       "L 18.109375 -20.796875 \r\n",
       "L 9.078125 -20.796875 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.390625 \r\n",
       "Q 20.953125 51.265625 25.265625 53.625 \r\n",
       "Q 29.59375 56 35.59375 56 \r\n",
       "Q 45.5625 56 51.78125 48.09375 \r\n",
       "Q 58.015625 40.1875 58.015625 27.296875 \r\n",
       "Q 58.015625 14.40625 51.78125 6.484375 \r\n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \r\n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \r\n",
       "Q 20.953125 3.328125 18.109375 8.203125 \r\n",
       "z\r\n",
       "M 48.6875 27.296875 \r\n",
       "Q 48.6875 37.203125 44.609375 42.84375 \r\n",
       "Q 40.53125 48.484375 33.40625 48.484375 \r\n",
       "Q 26.265625 48.484375 22.1875 42.84375 \r\n",
       "Q 18.109375 37.203125 18.109375 27.296875 \r\n",
       "Q 18.109375 17.390625 22.1875 11.75 \r\n",
       "Q 26.265625 6.109375 33.40625 6.109375 \r\n",
       "Q 40.53125 6.109375 44.609375 11.75 \r\n",
       "Q 48.6875 17.390625 48.6875 27.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-112\"/>\r\n",
       "      <path d=\"M 30.609375 48.390625 \r\n",
       "Q 23.390625 48.390625 19.1875 42.75 \r\n",
       "Q 14.984375 37.109375 14.984375 27.296875 \r\n",
       "Q 14.984375 17.484375 19.15625 11.84375 \r\n",
       "Q 23.34375 6.203125 30.609375 6.203125 \r\n",
       "Q 37.796875 6.203125 41.984375 11.859375 \r\n",
       "Q 46.1875 17.53125 46.1875 27.296875 \r\n",
       "Q 46.1875 37.015625 41.984375 42.703125 \r\n",
       "Q 37.796875 48.390625 30.609375 48.390625 \r\n",
       "z\r\n",
       "M 30.609375 56 \r\n",
       "Q 42.328125 56 49.015625 48.375 \r\n",
       "Q 55.71875 40.765625 55.71875 27.296875 \r\n",
       "Q 55.71875 13.875 49.015625 6.21875 \r\n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \r\n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \r\n",
       "Q 5.515625 13.875 5.515625 27.296875 \r\n",
       "Q 5.515625 40.765625 12.171875 48.375 \r\n",
       "Q 18.84375 56 30.609375 56 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-111\"/>\r\n",
       "      <path d=\"M 48.78125 52.59375 \r\n",
       "L 48.78125 44.1875 \r\n",
       "Q 44.96875 46.296875 41.140625 47.34375 \r\n",
       "Q 37.3125 48.390625 33.40625 48.390625 \r\n",
       "Q 24.65625 48.390625 19.8125 42.84375 \r\n",
       "Q 14.984375 37.3125 14.984375 27.296875 \r\n",
       "Q 14.984375 17.28125 19.8125 11.734375 \r\n",
       "Q 24.65625 6.203125 33.40625 6.203125 \r\n",
       "Q 37.3125 6.203125 41.140625 7.25 \r\n",
       "Q 44.96875 8.296875 48.78125 10.40625 \r\n",
       "L 48.78125 2.09375 \r\n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \r\n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \r\n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \r\n",
       "Q 5.515625 14.109375 5.515625 27.296875 \r\n",
       "Q 5.515625 40.671875 12.859375 48.328125 \r\n",
       "Q 20.21875 56 33.015625 56 \r\n",
       "Q 37.15625 56 41.109375 55.140625 \r\n",
       "Q 45.0625 54.296875 48.78125 52.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-99\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 75.984375 \r\n",
       "L 18.109375 75.984375 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-104\"/>\r\n",
       "      <path d=\"M 44.28125 53.078125 \r\n",
       "L 44.28125 44.578125 \r\n",
       "Q 40.484375 46.53125 36.375 47.5 \r\n",
       "Q 32.28125 48.484375 27.875 48.484375 \r\n",
       "Q 21.1875 48.484375 17.84375 46.4375 \r\n",
       "Q 14.5 44.390625 14.5 40.28125 \r\n",
       "Q 14.5 37.15625 16.890625 35.375 \r\n",
       "Q 19.28125 33.59375 26.515625 31.984375 \r\n",
       "L 29.59375 31.296875 \r\n",
       "Q 39.15625 29.25 43.1875 25.515625 \r\n",
       "Q 47.21875 21.78125 47.21875 15.09375 \r\n",
       "Q 47.21875 7.46875 41.1875 3.015625 \r\n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \r\n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \r\n",
       "Q 10.6875 0.296875 5.421875 2 \r\n",
       "L 5.421875 11.28125 \r\n",
       "Q 10.40625 8.6875 15.234375 7.390625 \r\n",
       "Q 20.0625 6.109375 24.8125 6.109375 \r\n",
       "Q 31.15625 6.109375 34.5625 8.28125 \r\n",
       "Q 37.984375 10.453125 37.984375 14.40625 \r\n",
       "Q 37.984375 18.0625 35.515625 20.015625 \r\n",
       "Q 33.0625 21.96875 24.703125 23.78125 \r\n",
       "L 21.578125 24.515625 \r\n",
       "Q 13.234375 26.265625 9.515625 29.90625 \r\n",
       "Q 5.8125 33.546875 5.8125 39.890625 \r\n",
       "Q 5.8125 47.609375 11.28125 51.796875 \r\n",
       "Q 16.75 56 26.8125 56 \r\n",
       "Q 31.78125 56 36.171875 55.265625 \r\n",
       "Q 40.578125 54.546875 44.28125 53.078125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-115\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(135.595313 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\r\n",
       "      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m7bc1750763\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m7bc1750763\" y=\"125.317107\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- $\\mathdefault{10^{-11}}$ -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-8722\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 129.116326)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(231.391602 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m7bc1750763\" y=\"99.51117\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- $\\mathdefault{10^{-8}}$ -->\r\n",
       "      <g transform=\"translate(25.278125 103.310389)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-56\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m7bc1750763\" y=\"73.705232\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- $\\mathdefault{10^{-5}}$ -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.796875 72.90625 \r\n",
       "L 49.515625 72.90625 \r\n",
       "L 49.515625 64.59375 \r\n",
       "L 19.828125 64.59375 \r\n",
       "L 19.828125 46.734375 \r\n",
       "Q 21.96875 47.46875 24.109375 47.828125 \r\n",
       "Q 26.265625 48.1875 28.421875 48.1875 \r\n",
       "Q 40.625 48.1875 47.75 41.5 \r\n",
       "Q 54.890625 34.8125 54.890625 23.390625 \r\n",
       "Q 54.890625 11.625 47.5625 5.09375 \r\n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \r\n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \r\n",
       "Q 12.796875 0.140625 7.71875 1.703125 \r\n",
       "L 7.71875 11.625 \r\n",
       "Q 12.109375 9.234375 16.796875 8.0625 \r\n",
       "Q 21.484375 6.890625 26.703125 6.890625 \r\n",
       "Q 35.15625 6.890625 40.078125 11.328125 \r\n",
       "Q 45.015625 15.765625 45.015625 23.390625 \r\n",
       "Q 45.015625 31 40.078125 35.4375 \r\n",
       "Q 35.15625 39.890625 26.703125 39.890625 \r\n",
       "Q 22.75 39.890625 18.8125 39.015625 \r\n",
       "Q 14.890625 38.140625 10.796875 36.28125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-53\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(25.278125 77.504451)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-53\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m7bc1750763\" y=\"47.899294\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- $\\mathdefault{10^{-2}}$ -->\r\n",
       "      <g transform=\"translate(25.278125 51.698513)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m7bc1750763\" y=\"22.093357\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- $\\mathdefault{10^{1}}$ -->\r\n",
       "      <g transform=\"translate(31.178125 25.892576)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_13\">\r\n",
       "     <!-- loss -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-108\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798438 84.807812)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-108\"/>\r\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_12\">\r\n",
       "    <path clip-path=\"url(#p70e585da65)\" d=\"M 64.655398 19.118444 \r\n",
       "L 66.448786 24.419184 \r\n",
       "L 68.242175 29.085184 \r\n",
       "L 70.035563 33.179526 \r\n",
       "L 71.828951 36.990705 \r\n",
       "L 73.62234 40.757131 \r\n",
       "L 75.415728 44.340979 \r\n",
       "L 77.209117 48.054459 \r\n",
       "L 79.002505 51.674744 \r\n",
       "L 80.795894 55.30831 \r\n",
       "L 82.589282 58.913457 \r\n",
       "L 84.38267 62.499477 \r\n",
       "L 86.176059 66.125104 \r\n",
       "L 87.969447 69.695831 \r\n",
       "L 89.762836 73.311471 \r\n",
       "L 91.556224 76.853163 \r\n",
       "L 93.349613 80.432435 \r\n",
       "L 95.143001 84.038857 \r\n",
       "L 96.936389 87.492883 \r\n",
       "L 98.729778 91.018965 \r\n",
       "L 100.523166 94.568095 \r\n",
       "L 102.316555 98.194576 \r\n",
       "L 104.109943 101.744562 \r\n",
       "L 105.903332 105.258589 \r\n",
       "L 107.69672 108.776946 \r\n",
       "L 109.490108 112.386685 \r\n",
       "L 111.283497 115.76727 \r\n",
       "L 113.076885 118.96469 \r\n",
       "L 114.870274 121.959092 \r\n",
       "L 116.663662 125.033281 \r\n",
       "L 118.457051 125.951307 \r\n",
       "L 120.250439 127.2141 \r\n",
       "L 122.043827 128.904828 \r\n",
       "L 123.837216 129.337559 \r\n",
       "L 125.630604 130.068999 \r\n",
       "L 127.423993 130.489257 \r\n",
       "L 129.217381 131.050423 \r\n",
       "L 131.01077 131.565304 \r\n",
       "L 132.804158 131.125592 \r\n",
       "L 134.597546 130.688392 \r\n",
       "L 136.390935 131.029754 \r\n",
       "L 138.184323 131.455863 \r\n",
       "L 139.977712 131.668317 \r\n",
       "L 141.7711 131.685545 \r\n",
       "L 143.564489 131.807569 \r\n",
       "L 145.357877 131.538431 \r\n",
       "L 147.151265 132.336057 \r\n",
       "L 148.944654 132.986367 \r\n",
       "L 150.738042 133.322695 \r\n",
       "L 152.531431 133.13555 \r\n",
       "L 154.324819 133.532928 \r\n",
       "L 156.118208 133.073288 \r\n",
       "L 157.911596 133.098375 \r\n",
       "L 159.704985 133.918394 \r\n",
       "L 161.498373 135.439878 \r\n",
       "L 163.291761 134.311498 \r\n",
       "L 165.08515 134.805695 \r\n",
       "L 166.878538 134.75104 \r\n",
       "L 168.671927 134.487366 \r\n",
       "L 170.465315 135.276739 \r\n",
       "L 172.258704 135.549396 \r\n",
       "L 174.052092 135.817518 \r\n",
       "L 175.84548 134.479021 \r\n",
       "L 177.638869 133.762616 \r\n",
       "L 179.432257 133.783516 \r\n",
       "L 181.225646 133.824376 \r\n",
       "L 183.019034 134.001925 \r\n",
       "L 184.812423 134.209433 \r\n",
       "L 186.605811 134.328953 \r\n",
       "L 188.399199 134.504063 \r\n",
       "L 190.192588 134.940454 \r\n",
       "L 191.985976 134.800493 \r\n",
       "L 193.779365 134.747208 \r\n",
       "L 195.572753 134.853111 \r\n",
       "L 197.366142 135.213803 \r\n",
       "L 199.15953 134.814751 \r\n",
       "L 200.952918 135.477958 \r\n",
       "L 202.746307 135.404279 \r\n",
       "L 204.539695 135.152255 \r\n",
       "L 206.333084 135.22514 \r\n",
       "L 208.126472 134.952904 \r\n",
       "L 209.919861 134.802594 \r\n",
       "L 211.713249 134.969075 \r\n",
       "L 213.506637 135.019738 \r\n",
       "L 215.300026 134.869122 \r\n",
       "L 217.093414 135.102849 \r\n",
       "L 218.886803 135.052851 \r\n",
       "L 220.680191 135.191113 \r\n",
       "L 222.47358 135.616473 \r\n",
       "L 224.266968 135.415778 \r\n",
       "L 226.060356 135.105127 \r\n",
       "L 227.853745 135.912336 \r\n",
       "L 229.647133 135.703587 \r\n",
       "L 231.440522 136.875301 \r\n",
       "L 233.23391 136.747037 \r\n",
       "L 235.027299 136.036575 \r\n",
       "L 236.820687 136.810529 \r\n",
       "L 238.614075 136.78186 \r\n",
       "L 240.407464 136.922727 \r\n",
       "L 242.200852 135.600436 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_13\">\r\n",
       "    <path clip-path=\"url(#p70e585da65)\" d=\"M 64.655398 13.399396 \r\n",
       "L 66.448786 13.380766 \r\n",
       "L 68.242175 13.379011 \r\n",
       "L 70.035563 13.378158 \r\n",
       "L 71.828951 13.377729 \r\n",
       "L 73.62234 13.377273 \r\n",
       "L 75.415728 13.377966 \r\n",
       "L 77.209117 13.378352 \r\n",
       "L 79.002505 13.378608 \r\n",
       "L 80.795894 13.378606 \r\n",
       "L 82.589282 13.378456 \r\n",
       "L 84.38267 13.378573 \r\n",
       "L 86.176059 13.378634 \r\n",
       "L 87.969447 13.378664 \r\n",
       "L 89.762836 13.378706 \r\n",
       "L 91.556224 13.378715 \r\n",
       "L 93.349613 13.378717 \r\n",
       "L 95.143001 13.378719 \r\n",
       "L 96.936389 13.378723 \r\n",
       "L 98.729778 13.378724 \r\n",
       "L 100.523166 13.378724 \r\n",
       "L 102.316555 13.378725 \r\n",
       "L 104.109943 13.378724 \r\n",
       "L 105.903332 13.378725 \r\n",
       "L 107.69672 13.378725 \r\n",
       "L 109.490108 13.378725 \r\n",
       "L 111.283497 13.378725 \r\n",
       "L 113.076885 13.378725 \r\n",
       "L 114.870274 13.378725 \r\n",
       "L 116.663662 13.378725 \r\n",
       "L 118.457051 13.378725 \r\n",
       "L 120.250439 13.378725 \r\n",
       "L 122.043827 13.378725 \r\n",
       "L 123.837216 13.378725 \r\n",
       "L 125.630604 13.378725 \r\n",
       "L 127.423993 13.378725 \r\n",
       "L 129.217381 13.378725 \r\n",
       "L 131.01077 13.378725 \r\n",
       "L 132.804158 13.378725 \r\n",
       "L 134.597546 13.378725 \r\n",
       "L 136.390935 13.378725 \r\n",
       "L 138.184323 13.378725 \r\n",
       "L 139.977712 13.378725 \r\n",
       "L 141.7711 13.378725 \r\n",
       "L 143.564489 13.378725 \r\n",
       "L 145.357877 13.378725 \r\n",
       "L 147.151265 13.378725 \r\n",
       "L 148.944654 13.378725 \r\n",
       "L 150.738042 13.378725 \r\n",
       "L 152.531431 13.378725 \r\n",
       "L 154.324819 13.378725 \r\n",
       "L 156.118208 13.378725 \r\n",
       "L 157.911596 13.378725 \r\n",
       "L 159.704985 13.378725 \r\n",
       "L 161.498373 13.378725 \r\n",
       "L 163.291761 13.378725 \r\n",
       "L 165.08515 13.378725 \r\n",
       "L 166.878538 13.378725 \r\n",
       "L 168.671927 13.378725 \r\n",
       "L 170.465315 13.378725 \r\n",
       "L 172.258704 13.378725 \r\n",
       "L 174.052092 13.378725 \r\n",
       "L 175.84548 13.378725 \r\n",
       "L 177.638869 13.378725 \r\n",
       "L 179.432257 13.378725 \r\n",
       "L 181.225646 13.378725 \r\n",
       "L 183.019034 13.378725 \r\n",
       "L 184.812423 13.378725 \r\n",
       "L 186.605811 13.378725 \r\n",
       "L 188.399199 13.378725 \r\n",
       "L 190.192588 13.378725 \r\n",
       "L 191.985976 13.378725 \r\n",
       "L 193.779365 13.378725 \r\n",
       "L 195.572753 13.378725 \r\n",
       "L 197.366142 13.378725 \r\n",
       "L 199.15953 13.378725 \r\n",
       "L 200.952918 13.378725 \r\n",
       "L 202.746307 13.378725 \r\n",
       "L 204.539695 13.378725 \r\n",
       "L 206.333084 13.378725 \r\n",
       "L 208.126472 13.378725 \r\n",
       "L 209.919861 13.378725 \r\n",
       "L 211.713249 13.378725 \r\n",
       "L 213.506637 13.378725 \r\n",
       "L 215.300026 13.378725 \r\n",
       "L 217.093414 13.378725 \r\n",
       "L 218.886803 13.378725 \r\n",
       "L 220.680191 13.378725 \r\n",
       "L 222.47358 13.378725 \r\n",
       "L 224.266968 13.378725 \r\n",
       "L 226.060356 13.378725 \r\n",
       "L 227.853745 13.378725 \r\n",
       "L 229.647133 13.378725 \r\n",
       "L 231.440522 13.378725 \r\n",
       "L 233.23391 13.378725 \r\n",
       "L 235.027299 13.378725 \r\n",
       "L 236.820687 13.378725 \r\n",
       "L 238.614075 13.378725 \r\n",
       "L 240.407464 13.378725 \r\n",
       "L 242.200852 13.378725 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 55.778125 143.1 \r\n",
       "L 55.778125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 251.078125 143.1 \r\n",
       "L 251.078125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 55.778125 143.1 \r\n",
       "L 251.078125 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 55.778125 7.2 \r\n",
       "L 251.078125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 188.803125 91.328125 \r\n",
       "L 244.078125 91.328125 \r\n",
       "Q 246.078125 91.328125 246.078125 89.328125 \r\n",
       "L 246.078125 60.971875 \r\n",
       "Q 246.078125 58.971875 244.078125 58.971875 \r\n",
       "L 188.803125 58.971875 \r\n",
       "Q 186.803125 58.971875 186.803125 60.971875 \r\n",
       "L 186.803125 89.328125 \r\n",
       "Q 186.803125 91.328125 188.803125 91.328125 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_14\">\r\n",
       "     <path d=\"M 190.803125 67.070312 \r\n",
       "L 210.803125 67.070312 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_15\"/>\r\n",
       "    <g id=\"text_14\">\r\n",
       "     <!-- train -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 18.3125 70.21875 \r\n",
       "L 18.3125 54.6875 \r\n",
       "L 36.8125 54.6875 \r\n",
       "L 36.8125 47.703125 \r\n",
       "L 18.3125 47.703125 \r\n",
       "L 18.3125 18.015625 \r\n",
       "Q 18.3125 11.328125 20.140625 9.421875 \r\n",
       "Q 21.96875 7.515625 27.59375 7.515625 \r\n",
       "L 36.8125 7.515625 \r\n",
       "L 36.8125 0 \r\n",
       "L 27.59375 0 \r\n",
       "Q 17.1875 0 13.234375 3.875 \r\n",
       "Q 9.28125 7.765625 9.28125 18.015625 \r\n",
       "L 9.28125 47.703125 \r\n",
       "L 2.6875 47.703125 \r\n",
       "L 2.6875 54.6875 \r\n",
       "L 9.28125 54.6875 \r\n",
       "L 9.28125 70.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-116\"/>\r\n",
       "      <path d=\"M 41.109375 46.296875 \r\n",
       "Q 39.59375 47.171875 37.8125 47.578125 \r\n",
       "Q 36.03125 48 33.890625 48 \r\n",
       "Q 26.265625 48 22.1875 43.046875 \r\n",
       "Q 18.109375 38.09375 18.109375 28.8125 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 20.953125 51.171875 25.484375 53.578125 \r\n",
       "Q 30.03125 56 36.53125 56 \r\n",
       "Q 37.453125 56 38.578125 55.875 \r\n",
       "Q 39.703125 55.765625 41.0625 55.515625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-114\"/>\r\n",
       "      <path d=\"M 34.28125 27.484375 \r\n",
       "Q 23.390625 27.484375 19.1875 25 \r\n",
       "Q 14.984375 22.515625 14.984375 16.5 \r\n",
       "Q 14.984375 11.71875 18.140625 8.90625 \r\n",
       "Q 21.296875 6.109375 26.703125 6.109375 \r\n",
       "Q 34.1875 6.109375 38.703125 11.40625 \r\n",
       "Q 43.21875 16.703125 43.21875 25.484375 \r\n",
       "L 43.21875 27.484375 \r\n",
       "z\r\n",
       "M 52.203125 31.203125 \r\n",
       "L 52.203125 0 \r\n",
       "L 43.21875 0 \r\n",
       "L 43.21875 8.296875 \r\n",
       "Q 40.140625 3.328125 35.546875 0.953125 \r\n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \r\n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \r\n",
       "Q 6 8.015625 6 15.921875 \r\n",
       "Q 6 25.140625 12.171875 29.828125 \r\n",
       "Q 18.359375 34.515625 30.609375 34.515625 \r\n",
       "L 43.21875 34.515625 \r\n",
       "L 43.21875 35.40625 \r\n",
       "Q 43.21875 41.609375 39.140625 45 \r\n",
       "Q 35.0625 48.390625 27.6875 48.390625 \r\n",
       "Q 23 48.390625 18.546875 47.265625 \r\n",
       "Q 14.109375 46.140625 10.015625 43.890625 \r\n",
       "L 10.015625 52.203125 \r\n",
       "Q 14.9375 54.109375 19.578125 55.046875 \r\n",
       "Q 24.21875 56 28.609375 56 \r\n",
       "Q 40.484375 56 46.34375 49.84375 \r\n",
       "Q 52.203125 43.703125 52.203125 31.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-97\"/>\r\n",
       "      <path d=\"M 9.421875 54.6875 \r\n",
       "L 18.40625 54.6875 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 64.59375 \r\n",
       "L 9.421875 64.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-105\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-110\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(218.803125 70.570312)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_16\">\r\n",
       "     <path d=\"M 190.803125 81.748437 \r\n",
       "L 210.803125 81.748437 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_17\"/>\r\n",
       "    <g id=\"text_15\">\r\n",
       "     <!-- test -->\r\n",
       "     <g transform=\"translate(218.803125 85.248437)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p70e585da65\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"55.778125\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_and_plot_pytorch(0)#参数设置为0，说明不启用权重衰减"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "graffitiCellId": "id_qclwxdh",
    "id": "3FAACA854B9545A8ADADDEB6EE17A680",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L2 norm of w: 0.1538914144039154\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\r\n",
       "<svg height=\"184.34149pt\" version=\"1.1\" viewBox=\"0 0 254.544602 184.34149\" width=\"254.544602pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 184.34149 \r\n",
       "L 254.544602 184.34149 \r\n",
       "L 254.544602 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 51.378125 146.78524 \r\n",
       "L 246.678125 146.78524 \r\n",
       "L 246.678125 10.88524 \r\n",
       "L 51.378125 10.88524 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"mf0f8959ec5\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"58.462009\" xlink:href=\"#mf0f8959ec5\" y=\"146.78524\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-48\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(55.280759 161.383678)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"94.329778\" xlink:href=\"#mf0f8959ec5\" y=\"146.78524\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 20 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-50\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(87.967278 161.383678)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"130.197546\" xlink:href=\"#mf0f8959ec5\" y=\"146.78524\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 40 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-52\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(123.835046 161.383678)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"166.065315\" xlink:href=\"#mf0f8959ec5\" y=\"146.78524\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 60 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 33.015625 40.375 \r\n",
       "Q 26.375 40.375 22.484375 35.828125 \r\n",
       "Q 18.609375 31.296875 18.609375 23.390625 \r\n",
       "Q 18.609375 15.53125 22.484375 10.953125 \r\n",
       "Q 26.375 6.390625 33.015625 6.390625 \r\n",
       "Q 39.65625 6.390625 43.53125 10.953125 \r\n",
       "Q 47.40625 15.53125 47.40625 23.390625 \r\n",
       "Q 47.40625 31.296875 43.53125 35.828125 \r\n",
       "Q 39.65625 40.375 33.015625 40.375 \r\n",
       "z\r\n",
       "M 52.59375 71.296875 \r\n",
       "L 52.59375 62.3125 \r\n",
       "Q 48.875 64.0625 45.09375 64.984375 \r\n",
       "Q 41.3125 65.921875 37.59375 65.921875 \r\n",
       "Q 27.828125 65.921875 22.671875 59.328125 \r\n",
       "Q 17.53125 52.734375 16.796875 39.40625 \r\n",
       "Q 19.671875 43.65625 24.015625 45.921875 \r\n",
       "Q 28.375 48.1875 33.59375 48.1875 \r\n",
       "Q 44.578125 48.1875 50.953125 41.515625 \r\n",
       "Q 57.328125 34.859375 57.328125 23.390625 \r\n",
       "Q 57.328125 12.15625 50.6875 5.359375 \r\n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \r\n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \r\n",
       "Q 6.984375 17.96875 6.984375 36.375 \r\n",
       "Q 6.984375 53.65625 15.1875 63.9375 \r\n",
       "Q 23.390625 74.21875 37.203125 74.21875 \r\n",
       "Q 40.921875 74.21875 44.703125 73.484375 \r\n",
       "Q 48.484375 72.75 52.59375 71.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-54\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(159.702815 161.383678)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-54\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"201.933084\" xlink:href=\"#mf0f8959ec5\" y=\"146.78524\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 80 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 34.625 \r\n",
       "Q 24.75 34.625 20.71875 30.859375 \r\n",
       "Q 16.703125 27.09375 16.703125 20.515625 \r\n",
       "Q 16.703125 13.921875 20.71875 10.15625 \r\n",
       "Q 24.75 6.390625 31.78125 6.390625 \r\n",
       "Q 38.8125 6.390625 42.859375 10.171875 \r\n",
       "Q 46.921875 13.96875 46.921875 20.515625 \r\n",
       "Q 46.921875 27.09375 42.890625 30.859375 \r\n",
       "Q 38.875 34.625 31.78125 34.625 \r\n",
       "z\r\n",
       "M 21.921875 38.8125 \r\n",
       "Q 15.578125 40.375 12.03125 44.71875 \r\n",
       "Q 8.5 49.078125 8.5 55.328125 \r\n",
       "Q 8.5 64.0625 14.71875 69.140625 \r\n",
       "Q 20.953125 74.21875 31.78125 74.21875 \r\n",
       "Q 42.671875 74.21875 48.875 69.140625 \r\n",
       "Q 55.078125 64.0625 55.078125 55.328125 \r\n",
       "Q 55.078125 49.078125 51.53125 44.71875 \r\n",
       "Q 48 40.375 41.703125 38.8125 \r\n",
       "Q 48.828125 37.15625 52.796875 32.3125 \r\n",
       "Q 56.78125 27.484375 56.78125 20.515625 \r\n",
       "Q 56.78125 9.90625 50.3125 4.234375 \r\n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.734375 -1.421875 13.25 4.234375 \r\n",
       "Q 6.78125 9.90625 6.78125 20.515625 \r\n",
       "Q 6.78125 27.484375 10.78125 32.3125 \r\n",
       "Q 14.796875 37.15625 21.921875 38.8125 \r\n",
       "z\r\n",
       "M 18.3125 54.390625 \r\n",
       "Q 18.3125 48.734375 21.84375 45.5625 \r\n",
       "Q 25.390625 42.390625 31.78125 42.390625 \r\n",
       "Q 38.140625 42.390625 41.71875 45.5625 \r\n",
       "Q 45.3125 48.734375 45.3125 54.390625 \r\n",
       "Q 45.3125 60.0625 41.71875 63.234375 \r\n",
       "Q 38.140625 66.40625 31.78125 66.40625 \r\n",
       "Q 25.390625 66.40625 21.84375 63.234375 \r\n",
       "Q 18.3125 60.0625 18.3125 54.390625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-56\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(195.570584 161.383678)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-56\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"237.800852\" xlink:href=\"#mf0f8959ec5\" y=\"146.78524\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-49\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(228.257102 161.383678)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_7\">\r\n",
       "     <!-- epochs -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 56.203125 29.59375 \r\n",
       "L 56.203125 25.203125 \r\n",
       "L 14.890625 25.203125 \r\n",
       "Q 15.484375 15.921875 20.484375 11.0625 \r\n",
       "Q 25.484375 6.203125 34.421875 6.203125 \r\n",
       "Q 39.59375 6.203125 44.453125 7.46875 \r\n",
       "Q 49.3125 8.734375 54.109375 11.28125 \r\n",
       "L 54.109375 2.78125 \r\n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \r\n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \r\n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \r\n",
       "Q 5.515625 13.8125 5.515625 26.8125 \r\n",
       "Q 5.515625 40.234375 12.765625 48.109375 \r\n",
       "Q 20.015625 56 32.328125 56 \r\n",
       "Q 43.359375 56 49.78125 48.890625 \r\n",
       "Q 56.203125 41.796875 56.203125 29.59375 \r\n",
       "z\r\n",
       "M 47.21875 32.234375 \r\n",
       "Q 47.125 39.59375 43.09375 43.984375 \r\n",
       "Q 39.0625 48.390625 32.421875 48.390625 \r\n",
       "Q 24.90625 48.390625 20.390625 44.140625 \r\n",
       "Q 15.875 39.890625 15.1875 32.171875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-101\"/>\r\n",
       "      <path d=\"M 18.109375 8.203125 \r\n",
       "L 18.109375 -20.796875 \r\n",
       "L 9.078125 -20.796875 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.390625 \r\n",
       "Q 20.953125 51.265625 25.265625 53.625 \r\n",
       "Q 29.59375 56 35.59375 56 \r\n",
       "Q 45.5625 56 51.78125 48.09375 \r\n",
       "Q 58.015625 40.1875 58.015625 27.296875 \r\n",
       "Q 58.015625 14.40625 51.78125 6.484375 \r\n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \r\n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \r\n",
       "Q 20.953125 3.328125 18.109375 8.203125 \r\n",
       "z\r\n",
       "M 48.6875 27.296875 \r\n",
       "Q 48.6875 37.203125 44.609375 42.84375 \r\n",
       "Q 40.53125 48.484375 33.40625 48.484375 \r\n",
       "Q 26.265625 48.484375 22.1875 42.84375 \r\n",
       "Q 18.109375 37.203125 18.109375 27.296875 \r\n",
       "Q 18.109375 17.390625 22.1875 11.75 \r\n",
       "Q 26.265625 6.109375 33.40625 6.109375 \r\n",
       "Q 40.53125 6.109375 44.609375 11.75 \r\n",
       "Q 48.6875 17.390625 48.6875 27.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-112\"/>\r\n",
       "      <path d=\"M 30.609375 48.390625 \r\n",
       "Q 23.390625 48.390625 19.1875 42.75 \r\n",
       "Q 14.984375 37.109375 14.984375 27.296875 \r\n",
       "Q 14.984375 17.484375 19.15625 11.84375 \r\n",
       "Q 23.34375 6.203125 30.609375 6.203125 \r\n",
       "Q 37.796875 6.203125 41.984375 11.859375 \r\n",
       "Q 46.1875 17.53125 46.1875 27.296875 \r\n",
       "Q 46.1875 37.015625 41.984375 42.703125 \r\n",
       "Q 37.796875 48.390625 30.609375 48.390625 \r\n",
       "z\r\n",
       "M 30.609375 56 \r\n",
       "Q 42.328125 56 49.015625 48.375 \r\n",
       "Q 55.71875 40.765625 55.71875 27.296875 \r\n",
       "Q 55.71875 13.875 49.015625 6.21875 \r\n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \r\n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \r\n",
       "Q 5.515625 13.875 5.515625 27.296875 \r\n",
       "Q 5.515625 40.765625 12.171875 48.375 \r\n",
       "Q 18.84375 56 30.609375 56 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-111\"/>\r\n",
       "      <path d=\"M 48.78125 52.59375 \r\n",
       "L 48.78125 44.1875 \r\n",
       "Q 44.96875 46.296875 41.140625 47.34375 \r\n",
       "Q 37.3125 48.390625 33.40625 48.390625 \r\n",
       "Q 24.65625 48.390625 19.8125 42.84375 \r\n",
       "Q 14.984375 37.3125 14.984375 27.296875 \r\n",
       "Q 14.984375 17.28125 19.8125 11.734375 \r\n",
       "Q 24.65625 6.203125 33.40625 6.203125 \r\n",
       "Q 37.3125 6.203125 41.140625 7.25 \r\n",
       "Q 44.96875 8.296875 48.78125 10.40625 \r\n",
       "L 48.78125 2.09375 \r\n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \r\n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \r\n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \r\n",
       "Q 5.515625 14.109375 5.515625 27.296875 \r\n",
       "Q 5.515625 40.671875 12.859375 48.328125 \r\n",
       "Q 20.21875 56 33.015625 56 \r\n",
       "Q 37.15625 56 41.109375 55.140625 \r\n",
       "Q 45.0625 54.296875 48.78125 52.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-99\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 75.984375 \r\n",
       "L 18.109375 75.984375 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-104\"/>\r\n",
       "      <path d=\"M 44.28125 53.078125 \r\n",
       "L 44.28125 44.578125 \r\n",
       "Q 40.484375 46.53125 36.375 47.5 \r\n",
       "Q 32.28125 48.484375 27.875 48.484375 \r\n",
       "Q 21.1875 48.484375 17.84375 46.4375 \r\n",
       "Q 14.5 44.390625 14.5 40.28125 \r\n",
       "Q 14.5 37.15625 16.890625 35.375 \r\n",
       "Q 19.28125 33.59375 26.515625 31.984375 \r\n",
       "L 29.59375 31.296875 \r\n",
       "Q 39.15625 29.25 43.1875 25.515625 \r\n",
       "Q 47.21875 21.78125 47.21875 15.09375 \r\n",
       "Q 47.21875 7.46875 41.1875 3.015625 \r\n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \r\n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \r\n",
       "Q 10.6875 0.296875 5.421875 2 \r\n",
       "L 5.421875 11.28125 \r\n",
       "Q 10.40625 8.6875 15.234375 7.390625 \r\n",
       "Q 20.0625 6.109375 24.8125 6.109375 \r\n",
       "Q 31.15625 6.109375 34.5625 8.28125 \r\n",
       "Q 37.984375 10.453125 37.984375 14.40625 \r\n",
       "Q 37.984375 18.0625 35.515625 20.015625 \r\n",
       "Q 33.0625 21.96875 24.703125 23.78125 \r\n",
       "L 21.578125 24.515625 \r\n",
       "Q 13.234375 26.265625 9.515625 29.90625 \r\n",
       "Q 5.8125 33.546875 5.8125 39.890625 \r\n",
       "Q 5.8125 47.609375 11.28125 51.796875 \r\n",
       "Q 16.75 56 26.8125 56 \r\n",
       "Q 31.78125 56 36.171875 55.265625 \r\n",
       "Q 40.578125 54.546875 44.28125 53.078125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-115\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(131.195312 175.061803)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\r\n",
       "      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m1029ee7be0\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m1029ee7be0\" y=\"132.470737\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- $\\mathdefault{10^{-2}}$ -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-8722\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 136.269955)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m1029ee7be0\" y=\"102.102857\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- $\\mathdefault{10^{-1}}$ -->\r\n",
       "      <g transform=\"translate(20.878125 105.902076)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m1029ee7be0\" y=\"71.734978\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- $\\mathdefault{10^{0}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 75.534196)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m1029ee7be0\" y=\"41.367098\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- $\\mathdefault{10^{1}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 45.166317)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m1029ee7be0\" y=\"10.999219\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- $\\mathdefault{10^{2}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 14.798437)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_6\">\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -2 0 \r\n",
       "\" id=\"meb5c91775e\" style=\"stroke:#000000;stroke-width:0.6;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"144.555331\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_7\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"141.612379\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_8\">\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"139.207813\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_9\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"137.174781\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_10\">\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"135.413688\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_11\">\r\n",
       "     <g id=\"line2d_17\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"133.860295\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_12\">\r\n",
       "     <g id=\"line2d_18\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"123.329094\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_13\">\r\n",
       "     <g id=\"line2d_19\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"117.981576\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_14\">\r\n",
       "     <g id=\"line2d_20\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"114.187451\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_15\">\r\n",
       "     <g id=\"line2d_21\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"111.2445\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_16\">\r\n",
       "     <g id=\"line2d_22\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"108.839933\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_17\">\r\n",
       "     <g id=\"line2d_23\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"106.806901\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_18\">\r\n",
       "     <g id=\"line2d_24\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"105.045809\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_19\">\r\n",
       "     <g id=\"line2d_25\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"103.492415\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_20\">\r\n",
       "     <g id=\"line2d_26\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"92.961214\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_21\">\r\n",
       "     <g id=\"line2d_27\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"87.613696\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_22\">\r\n",
       "     <g id=\"line2d_28\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"83.819572\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_23\">\r\n",
       "     <g id=\"line2d_29\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"80.87662\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_24\">\r\n",
       "     <g id=\"line2d_30\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"78.472054\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_25\">\r\n",
       "     <g id=\"line2d_31\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"76.439022\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_26\">\r\n",
       "     <g id=\"line2d_32\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"74.677929\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_27\">\r\n",
       "     <g id=\"line2d_33\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"73.124536\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_28\">\r\n",
       "     <g id=\"line2d_34\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"62.593335\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_29\">\r\n",
       "     <g id=\"line2d_35\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"57.245817\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_30\">\r\n",
       "     <g id=\"line2d_36\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"53.451692\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_31\">\r\n",
       "     <g id=\"line2d_37\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"50.508741\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_32\">\r\n",
       "     <g id=\"line2d_38\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"48.104174\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_33\">\r\n",
       "     <g id=\"line2d_39\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"46.071142\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_34\">\r\n",
       "     <g id=\"line2d_40\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"44.31005\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_35\">\r\n",
       "     <g id=\"line2d_41\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"42.756656\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_36\">\r\n",
       "     <g id=\"line2d_42\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"32.225456\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_37\">\r\n",
       "     <g id=\"line2d_43\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"26.877937\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_38\">\r\n",
       "     <g id=\"line2d_44\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"23.083813\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_39\">\r\n",
       "     <g id=\"line2d_45\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"20.140861\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_40\">\r\n",
       "     <g id=\"line2d_46\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"17.736295\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_41\">\r\n",
       "     <g id=\"line2d_47\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"15.703263\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_42\">\r\n",
       "     <g id=\"line2d_48\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"13.94217\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_43\">\r\n",
       "     <g id=\"line2d_49\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#meb5c91775e\" y=\"12.388777\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_13\">\r\n",
       "     <!-- loss -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-108\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798437 88.493053)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-108\"/>\r\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_50\">\r\n",
       "    <path clip-path=\"url(#pe1fc0b31d6)\" d=\"M 60.255398 38.98133 \r\n",
       "L 62.048786 61.76867 \r\n",
       "L 63.842175 78.947754 \r\n",
       "L 65.635563 91.498827 \r\n",
       "L 67.428951 100.036563 \r\n",
       "L 69.22234 104.350095 \r\n",
       "L 71.015728 107.024578 \r\n",
       "L 72.809117 108.016377 \r\n",
       "L 74.602505 108.505857 \r\n",
       "L 76.395894 108.614677 \r\n",
       "L 78.189282 109.792589 \r\n",
       "L 79.98267 109.347606 \r\n",
       "L 81.776059 109.911467 \r\n",
       "L 83.569447 110.728902 \r\n",
       "L 85.362836 110.859123 \r\n",
       "L 87.156224 111.265955 \r\n",
       "L 88.949613 111.543438 \r\n",
       "L 90.743001 112.313931 \r\n",
       "L 92.536389 112.268865 \r\n",
       "L 94.329778 113.047345 \r\n",
       "L 96.123166 113.176059 \r\n",
       "L 97.916555 113.599572 \r\n",
       "L 99.709943 114.681832 \r\n",
       "L 101.503332 114.620227 \r\n",
       "L 103.29672 114.646691 \r\n",
       "L 105.090108 114.485146 \r\n",
       "L 106.883497 115.495126 \r\n",
       "L 108.676885 115.364667 \r\n",
       "L 110.470274 116.196556 \r\n",
       "L 112.263662 116.3868 \r\n",
       "L 114.057051 117.167107 \r\n",
       "L 115.850439 117.094351 \r\n",
       "L 117.643827 117.176707 \r\n",
       "L 119.437216 117.811641 \r\n",
       "L 121.230604 118.438139 \r\n",
       "L 123.023993 118.634514 \r\n",
       "L 124.817381 119.159755 \r\n",
       "L 126.61077 119.52395 \r\n",
       "L 128.404158 119.511827 \r\n",
       "L 130.197546 120.030752 \r\n",
       "L 131.990935 120.551817 \r\n",
       "L 133.784323 120.387338 \r\n",
       "L 135.577712 120.833705 \r\n",
       "L 137.3711 121.649856 \r\n",
       "L 139.164489 121.842783 \r\n",
       "L 140.957877 122.462421 \r\n",
       "L 142.751265 121.973272 \r\n",
       "L 144.544654 122.393588 \r\n",
       "L 146.338042 122.71834 \r\n",
       "L 148.131431 123.381605 \r\n",
       "L 149.924819 123.644772 \r\n",
       "L 151.718208 124.542851 \r\n",
       "L 153.511596 123.671643 \r\n",
       "L 155.304985 124.381044 \r\n",
       "L 157.098373 125.722178 \r\n",
       "L 158.891761 126.36699 \r\n",
       "L 160.68515 126.717789 \r\n",
       "L 162.478538 126.77283 \r\n",
       "L 164.271927 126.809782 \r\n",
       "L 166.065315 127.271632 \r\n",
       "L 167.858704 128.380691 \r\n",
       "L 169.652092 128.320902 \r\n",
       "L 171.44548 127.771392 \r\n",
       "L 173.238869 128.469319 \r\n",
       "L 175.032257 128.994406 \r\n",
       "L 176.825646 128.351399 \r\n",
       "L 178.619034 129.176071 \r\n",
       "L 180.412423 129.917995 \r\n",
       "L 182.205811 129.472468 \r\n",
       "L 183.999199 130.006288 \r\n",
       "L 185.792588 130.190909 \r\n",
       "L 187.585976 131.173811 \r\n",
       "L 189.379365 131.710554 \r\n",
       "L 191.172753 131.071762 \r\n",
       "L 192.966142 131.494102 \r\n",
       "L 194.75953 132.741711 \r\n",
       "L 196.552918 134.171224 \r\n",
       "L 198.346307 133.398062 \r\n",
       "L 200.139695 133.79421 \r\n",
       "L 201.933084 133.208413 \r\n",
       "L 203.726472 133.864749 \r\n",
       "L 205.519861 133.996077 \r\n",
       "L 207.313249 134.70661 \r\n",
       "L 209.106637 135.321234 \r\n",
       "L 210.900026 135.4819 \r\n",
       "L 212.693414 136.693776 \r\n",
       "L 214.486803 136.095648 \r\n",
       "L 216.280191 137.346001 \r\n",
       "L 218.07358 137.329921 \r\n",
       "L 219.866968 137.639557 \r\n",
       "L 221.660356 137.231 \r\n",
       "L 223.453745 137.780471 \r\n",
       "L 225.247133 137.619636 \r\n",
       "L 227.040522 139.23472 \r\n",
       "L 228.83391 140.137543 \r\n",
       "L 230.627299 139.737647 \r\n",
       "L 232.420687 139.64704 \r\n",
       "L 234.214075 139.317024 \r\n",
       "L 236.007464 140.607967 \r\n",
       "L 237.800852 140.270232 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_51\">\r\n",
       "    <path clip-path=\"url(#pe1fc0b31d6)\" d=\"M 60.255398 17.062513 \r\n",
       "L 62.048786 21.444216 \r\n",
       "L 63.842175 25.80288 \r\n",
       "L 65.635563 30.109339 \r\n",
       "L 67.428951 34.291519 \r\n",
       "L 69.22234 38.324387 \r\n",
       "L 71.015728 42.116529 \r\n",
       "L 72.809117 45.682901 \r\n",
       "L 74.602505 48.940192 \r\n",
       "L 76.395894 51.890119 \r\n",
       "L 78.189282 54.476402 \r\n",
       "L 79.98267 56.751858 \r\n",
       "L 81.776059 58.690875 \r\n",
       "L 83.569447 60.278577 \r\n",
       "L 85.362836 61.661152 \r\n",
       "L 87.156224 62.825281 \r\n",
       "L 88.949613 63.81356 \r\n",
       "L 90.743001 64.578303 \r\n",
       "L 92.536389 65.331372 \r\n",
       "L 94.329778 65.989702 \r\n",
       "L 96.123166 66.567924 \r\n",
       "L 97.916555 67.06589 \r\n",
       "L 99.709943 67.492149 \r\n",
       "L 101.503332 67.977059 \r\n",
       "L 103.29672 68.389401 \r\n",
       "L 105.090108 68.830791 \r\n",
       "L 106.883497 69.248816 \r\n",
       "L 108.676885 69.652539 \r\n",
       "L 110.470274 69.992821 \r\n",
       "L 112.263662 70.392052 \r\n",
       "L 114.057051 70.733976 \r\n",
       "L 115.850439 71.123067 \r\n",
       "L 117.643827 71.517464 \r\n",
       "L 119.437216 71.86551 \r\n",
       "L 121.230604 72.206665 \r\n",
       "L 123.023993 72.593042 \r\n",
       "L 124.817381 72.975832 \r\n",
       "L 126.61077 73.339493 \r\n",
       "L 128.404158 73.650617 \r\n",
       "L 130.197546 74.011213 \r\n",
       "L 131.990935 74.42171 \r\n",
       "L 133.784323 74.789826 \r\n",
       "L 135.577712 75.154251 \r\n",
       "L 137.3711 75.496019 \r\n",
       "L 139.164489 75.886719 \r\n",
       "L 140.957877 76.230559 \r\n",
       "L 142.751265 76.556973 \r\n",
       "L 144.544654 76.89189 \r\n",
       "L 146.338042 77.263882 \r\n",
       "L 148.131431 77.611507 \r\n",
       "L 149.924819 78.036749 \r\n",
       "L 151.718208 78.336181 \r\n",
       "L 153.511596 78.716101 \r\n",
       "L 155.304985 79.108175 \r\n",
       "L 157.098373 79.443427 \r\n",
       "L 158.891761 79.770766 \r\n",
       "L 160.68515 80.16584 \r\n",
       "L 162.478538 80.516254 \r\n",
       "L 164.271927 80.905297 \r\n",
       "L 166.065315 81.222801 \r\n",
       "L 167.858704 81.56609 \r\n",
       "L 169.652092 81.917748 \r\n",
       "L 171.44548 82.352687 \r\n",
       "L 173.238869 82.693843 \r\n",
       "L 175.032257 83.027463 \r\n",
       "L 176.825646 83.449457 \r\n",
       "L 178.619034 83.777505 \r\n",
       "L 180.412423 84.163865 \r\n",
       "L 182.205811 84.549542 \r\n",
       "L 183.999199 84.885771 \r\n",
       "L 185.792588 85.241259 \r\n",
       "L 187.585976 85.583633 \r\n",
       "L 189.379365 85.919879 \r\n",
       "L 191.172753 86.230828 \r\n",
       "L 192.966142 86.593321 \r\n",
       "L 194.75953 87.033426 \r\n",
       "L 196.552918 87.350579 \r\n",
       "L 198.346307 87.742362 \r\n",
       "L 200.139695 88.041785 \r\n",
       "L 201.933084 88.402939 \r\n",
       "L 203.726472 88.773313 \r\n",
       "L 205.519861 89.129121 \r\n",
       "L 207.313249 89.488181 \r\n",
       "L 209.106637 89.828734 \r\n",
       "L 210.900026 90.198509 \r\n",
       "L 212.693414 90.580911 \r\n",
       "L 214.486803 90.940047 \r\n",
       "L 216.280191 91.260105 \r\n",
       "L 218.07358 91.658956 \r\n",
       "L 219.866968 91.922595 \r\n",
       "L 221.660356 92.36077 \r\n",
       "L 223.453745 92.704859 \r\n",
       "L 225.247133 93.101315 \r\n",
       "L 227.040522 93.395734 \r\n",
       "L 228.83391 93.732993 \r\n",
       "L 230.627299 94.090293 \r\n",
       "L 232.420687 94.506661 \r\n",
       "L 234.214075 94.799029 \r\n",
       "L 236.007464 95.195112 \r\n",
       "L 237.800852 95.567561 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 51.378125 146.78524 \r\n",
       "L 51.378125 10.88524 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 246.678125 146.78524 \r\n",
       "L 246.678125 10.88524 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 51.378125 146.78524 \r\n",
       "L 246.678125 146.78524 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 51.378125 10.88524 \r\n",
       "L 246.678125 10.88524 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 184.403125 48.24149 \r\n",
       "L 239.678125 48.24149 \r\n",
       "Q 241.678125 48.24149 241.678125 46.24149 \r\n",
       "L 241.678125 17.88524 \r\n",
       "Q 241.678125 15.88524 239.678125 15.88524 \r\n",
       "L 184.403125 15.88524 \r\n",
       "Q 182.403125 15.88524 182.403125 17.88524 \r\n",
       "L 182.403125 46.24149 \r\n",
       "Q 182.403125 48.24149 184.403125 48.24149 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_52\">\r\n",
       "     <path d=\"M 186.403125 23.983678 \r\n",
       "L 206.403125 23.983678 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_53\"/>\r\n",
       "    <g id=\"text_14\">\r\n",
       "     <!-- train -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 18.3125 70.21875 \r\n",
       "L 18.3125 54.6875 \r\n",
       "L 36.8125 54.6875 \r\n",
       "L 36.8125 47.703125 \r\n",
       "L 18.3125 47.703125 \r\n",
       "L 18.3125 18.015625 \r\n",
       "Q 18.3125 11.328125 20.140625 9.421875 \r\n",
       "Q 21.96875 7.515625 27.59375 7.515625 \r\n",
       "L 36.8125 7.515625 \r\n",
       "L 36.8125 0 \r\n",
       "L 27.59375 0 \r\n",
       "Q 17.1875 0 13.234375 3.875 \r\n",
       "Q 9.28125 7.765625 9.28125 18.015625 \r\n",
       "L 9.28125 47.703125 \r\n",
       "L 2.6875 47.703125 \r\n",
       "L 2.6875 54.6875 \r\n",
       "L 9.28125 54.6875 \r\n",
       "L 9.28125 70.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-116\"/>\r\n",
       "      <path d=\"M 41.109375 46.296875 \r\n",
       "Q 39.59375 47.171875 37.8125 47.578125 \r\n",
       "Q 36.03125 48 33.890625 48 \r\n",
       "Q 26.265625 48 22.1875 43.046875 \r\n",
       "Q 18.109375 38.09375 18.109375 28.8125 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 20.953125 51.171875 25.484375 53.578125 \r\n",
       "Q 30.03125 56 36.53125 56 \r\n",
       "Q 37.453125 56 38.578125 55.875 \r\n",
       "Q 39.703125 55.765625 41.0625 55.515625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-114\"/>\r\n",
       "      <path d=\"M 34.28125 27.484375 \r\n",
       "Q 23.390625 27.484375 19.1875 25 \r\n",
       "Q 14.984375 22.515625 14.984375 16.5 \r\n",
       "Q 14.984375 11.71875 18.140625 8.90625 \r\n",
       "Q 21.296875 6.109375 26.703125 6.109375 \r\n",
       "Q 34.1875 6.109375 38.703125 11.40625 \r\n",
       "Q 43.21875 16.703125 43.21875 25.484375 \r\n",
       "L 43.21875 27.484375 \r\n",
       "z\r\n",
       "M 52.203125 31.203125 \r\n",
       "L 52.203125 0 \r\n",
       "L 43.21875 0 \r\n",
       "L 43.21875 8.296875 \r\n",
       "Q 40.140625 3.328125 35.546875 0.953125 \r\n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \r\n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \r\n",
       "Q 6 8.015625 6 15.921875 \r\n",
       "Q 6 25.140625 12.171875 29.828125 \r\n",
       "Q 18.359375 34.515625 30.609375 34.515625 \r\n",
       "L 43.21875 34.515625 \r\n",
       "L 43.21875 35.40625 \r\n",
       "Q 43.21875 41.609375 39.140625 45 \r\n",
       "Q 35.0625 48.390625 27.6875 48.390625 \r\n",
       "Q 23 48.390625 18.546875 47.265625 \r\n",
       "Q 14.109375 46.140625 10.015625 43.890625 \r\n",
       "L 10.015625 52.203125 \r\n",
       "Q 14.9375 54.109375 19.578125 55.046875 \r\n",
       "Q 24.21875 56 28.609375 56 \r\n",
       "Q 40.484375 56 46.34375 49.84375 \r\n",
       "Q 52.203125 43.703125 52.203125 31.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-97\"/>\r\n",
       "      <path d=\"M 9.421875 54.6875 \r\n",
       "L 18.40625 54.6875 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 64.59375 \r\n",
       "L 9.421875 64.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-105\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-110\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(214.403125 27.483678)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_54\">\r\n",
       "     <path d=\"M 186.403125 38.661803 \r\n",
       "L 206.403125 38.661803 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_55\"/>\r\n",
       "    <g id=\"text_15\">\r\n",
       "     <!-- test -->\r\n",
       "     <g transform=\"translate(214.403125 42.161803)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"pe1fc0b31d6\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"51.378125\" y=\"10.88524\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_and_plot_pytorch(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_c8j5g7h",
    "id": "3547FD2B0E60465BB11AA6BADF4AC288",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# 丢弃法\n",
    "\n",
    "多层感知机中神经网络图描述了一个单隐藏层的多层感知机。其中输入个数为4，隐藏单元个数为5，且隐藏单元$h_i$（$i=1, \\ldots, 5$）的计算表达式为\n",
    "\n",
    "\n",
    "$$\n",
    " h_i = \\phi\\left(x_1 w_{1i} + x_2 w_{2i} + x_3 w_{3i} + x_4 w_{4i} + b_i\\right) \n",
    "$$\n",
    "\n",
    "\n",
    "这里$\\phi$是激活函数，$x_1, \\ldots, x_4$是输入，隐藏单元$i$的权重参数为$w_{1i}, \\ldots, w_{4i}$，偏差参数为$b_i$。当对该隐藏层使用丢弃法时，该层的隐藏单元将有一定概率被丢弃掉。设丢弃概率为$p$，那么有$p$的概率$h_i$会被清零，有$1-p$的概率$h_i$会除以$1-p$做拉伸。丢弃概率是丢弃法的超参数。具体来说，设随机变量$\\xi_i$为0和1的概率分别为$p$和$1-p$。使用丢弃法时我们计算新的隐藏单元$h_i'$\n",
    "\n",
    "\n",
    "$$\n",
    " h_i' = \\frac{\\xi_i}{1-p} h_i \n",
    "$$\n",
    "\n",
    "\n",
    "由于$E(\\xi_i) = 1-p$，因此\n",
    "\n",
    "\n",
    "$$\n",
    " E(h_i') = \\frac{E(\\xi_i)}{1-p}h_i = h_i \n",
    "$$\n",
    "\n",
    "\n",
    "即丢弃法不改变其输入的期望值。让我们对之前多层感知机的神经网络中的隐藏层使用丢弃法，一种可能的结果如图所示，其中$h_2$和$h_5$被清零。这时输出值的计算不再依赖$h_2$和$h_5$，在反向传播时，与这两个隐藏单元相关的权重的梯度均为0。由于在训练中隐藏层神经元的丢弃是随机的，即$h_1, \\ldots, h_5$都有可能被清零，输出层的计算无法过度依赖$h_1, \\ldots, h_5$中的任一个，从而在训练模型时起到正则化的作用，并可以用来应对过拟合。在测试模型时，我们为了拿到更加确定性的结果，一般不使用丢弃法\n",
    "\n",
    "![Image Name](https://cdn.kesci.com/upload/image/q5jd69in3m.png?imageView2/0/w/960/h/960)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_698weeu",
    "id": "53FBE46C03C040D0A123E957865CEC6A",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 丢弃法从零开始的实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "graffitiCellId": "id_0cw2xsh",
    "id": "30FE9592111F415A8C913AC04AD9BF9A",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.2.0\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import numpy as np\n",
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "import d2lzh1981 as d2l\n",
    "\n",
    "print(torch.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "graffitiCellId": "id_7mt4bcl",
    "id": "C6790DFD4650485AB8315E04E12D4C74",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def dropout(X, drop_prob):\n",
    "    X = X.float()\n",
    "    assert 0 <= drop_prob <= 1\n",
    "    keep_prob = 1 - drop_prob\n",
    "    # 这种情况下把全部元素都丢弃\n",
    "    if keep_prob == 0:\n",
    "        return torch.zeros_like(X)\n",
    "    mask = (torch.rand(X.shape) < keep_prob).float()\n",
    "    \n",
    "    return mask * X / keep_prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "graffitiCellId": "id_ek4ildz",
    "id": "F961F801B5CB434683F8F82FD79C8F23",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.],\n",
       "        [ 8.,  9., 10., 11., 12., 13., 14., 15.]])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = torch.arange(16).view(2, 8)\n",
    "dropout(X, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "graffitiCellId": "id_i67qnbo",
    "id": "A2563ED1062D48728E6AA868A3A8C3F4",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  2.,  4.,  0.,  8., 10.,  0., 14.],\n",
       "        [16., 18.,  0.,  0.,  0., 26.,  0., 30.]])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dropout(X, 0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "graffitiCellId": "id_gwsj0hd",
    "id": "56DF0D9E4EE84D04A6CCE5180FF9D164",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0.]])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dropout(X, 1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "graffitiCellId": "id_w81sasb",
    "id": "E3ECEB59A5F944898BE2C5ABEB513208",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 参数的初始化\n",
    "num_inputs, num_outputs, num_hiddens1, num_hiddens2 = 784, 10, 256, 256\n",
    "\n",
    "W1 = torch.tensor(np.random.normal(0, 0.01, size=(num_inputs, num_hiddens1)), dtype=torch.float, requires_grad=True)\n",
    "b1 = torch.zeros(num_hiddens1, requires_grad=True)\n",
    "W2 = torch.tensor(np.random.normal(0, 0.01, size=(num_hiddens1, num_hiddens2)), dtype=torch.float, requires_grad=True)\n",
    "b2 = torch.zeros(num_hiddens2, requires_grad=True)\n",
    "W3 = torch.tensor(np.random.normal(0, 0.01, size=(num_hiddens2, num_outputs)), dtype=torch.float, requires_grad=True)\n",
    "b3 = torch.zeros(num_outputs, requires_grad=True)\n",
    "\n",
    "params = [W1, b1, W2, b2, W3, b3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "graffitiCellId": "id_d24i59y",
    "id": "C3C7D8DC21FA471AA4AF11DFAC41D7E5",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "drop_prob1, drop_prob2 = 0.2, 0.5\n",
    "\n",
    "def net(X, is_training=True):\n",
    "    X = X.view(-1, num_inputs)\n",
    "    H1 = (torch.matmul(X, W1) + b1).relu()\n",
    "    if is_training:  # 只在训练模型时使用丢弃法\n",
    "        H1 = dropout(H1, drop_prob1)  # 在第一层全连接后添加丢弃层\n",
    "    H2 = (torch.matmul(H1, W2) + b2).relu()\n",
    "    if is_training:\n",
    "        H2 = dropout(H2, drop_prob2)  # 在第二层全连接后添加丢弃层\n",
    "    return torch.matmul(H2, W3) + b3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "graffitiCellId": "id_ticmy78",
    "id": "73C1F705FAA14F4390B8D24CCF9E5673",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def evaluate_accuracy(data_iter, net):\n",
    "    acc_sum, n = 0.0, 0\n",
    "    for X, y in data_iter:\n",
    "        if isinstance(net, torch.nn.Module):\n",
    "            net.eval() # 评估模式, 这会关闭dropout\n",
    "            acc_sum += (net(X).argmax(dim=1) == y).float().sum().item()\n",
    "            net.train() # 改回训练模式\n",
    "        else: # 自定义的模型\n",
    "            if('is_training' in net.__code__.co_varnames): # 如果有is_training这个参数\n",
    "                # 将is_training设置成False\n",
    "                acc_sum += (net(X, is_training=False).argmax(dim=1) == y).float().sum().item() \n",
    "            else:\n",
    "                acc_sum += (net(X).argmax(dim=1) == y).float().sum().item() \n",
    "        n += y.shape[0]\n",
    "    return acc_sum / n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "graffitiCellId": "id_ryzllq8",
    "id": "B493E72017B54831A80790EE0AA3DB2C",
    "jupyter": {},
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 1, loss 0.0045, train acc 0.554, test acc 0.749\n",
      "epoch 2, loss 0.0023, train acc 0.787, test acc 0.790\n",
      "epoch 3, loss 0.0020, train acc 0.820, test acc 0.737\n",
      "epoch 4, loss 0.0017, train acc 0.839, test acc 0.828\n",
      "epoch 5, loss 0.0016, train acc 0.848, test acc 0.837\n"
     ]
    }
   ],
   "source": [
    "num_epochs, lr, batch_size = 5, 100.0, 256  # 这里的学习率设置的很大，原因与之前相同。\n",
    "loss = torch.nn.CrossEntropyLoss()\n",
    "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size, root='C:/Users/Administrator/Datasets/FashionMNIST')\n",
    "d2l.train_ch3(\n",
    "    net,\n",
    "    train_iter,\n",
    "    test_iter,\n",
    "    loss,\n",
    "    num_epochs,\n",
    "    batch_size,\n",
    "    params,\n",
    "    lr)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_ar5qzrs",
    "id": "99EEA36EC08E471E8EC87F0AF6FCEF34",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 简洁实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "graffitiCellId": "id_t8mnig0",
    "id": "28E345EC41EB48D8876B6A15A3E18FBD",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "net = nn.Sequential(\n",
    "        d2l.FlattenLayer(),\n",
    "        nn.Linear(num_inputs, num_hiddens1),\n",
    "        nn.ReLU(),\n",
    "        nn.Dropout(drop_prob1),\n",
    "        nn.Linear(num_hiddens1, num_hiddens2), \n",
    "        nn.ReLU(),\n",
    "        nn.Dropout(drop_prob2),\n",
    "        nn.Linear(num_hiddens2, 10)\n",
    "        )\n",
    "\n",
    "for param in net.parameters():\n",
    "    nn.init.normal_(param, mean=0, std=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "graffitiCellId": "id_iba0hj1",
    "id": "2EDA1430541148BB863DC9071ED92323",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 1, loss 0.0045, train acc 0.560, test acc 0.788\n",
      "epoch 2, loss 0.0022, train acc 0.792, test acc 0.796\n",
      "epoch 3, loss 0.0019, train acc 0.822, test acc 0.837\n",
      "epoch 4, loss 0.0017, train acc 0.836, test acc 0.831\n",
      "epoch 5, loss 0.0016, train acc 0.846, test acc 0.825\n"
     ]
    }
   ],
   "source": [
    "optimizer = torch.optim.SGD(net.parameters(), lr=0.5)\n",
    "d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size, None, None, optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_m2ujou7",
    "id": "88258E5A8E1B4FCAA576FB7FB1AAC42C",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# 总结\n",
    "\n",
    "- 欠拟合现象：模型无法达到一个较低的误差\n",
    "    \n",
    "- 过拟合现象：训练误差较低但是泛化误差依然较高，二者相差较大\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
